Image decoding device and method for setting information for controlling decoding of coded data

ABSTRACT

The present disclosure relates to an image decoding device capable of recognizing performance necessary for decoding more accurately and a method. Coded data of image data and decoding load definition information for defining a magnitude of a load of a decoding process of a partial region of an image of the image data are acquired; decoding of the acquired coded data is controlled based on the acquired decoding load definition information; and the acquired coded data is decoded according to the controlling. The present disclosure can be applied to an information processing device such as an image coding device that scalably codes image data or an image decoding device that decodes encoded data obtained by scalably coding image data.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/902,761 (filed on Jan. 4, 2016), which is a National Stage PatentApplication of PCT International Patent Application No.PCT/JP2014/068259 (filed on Jul. 9, 2014) under 35 U.S.C. § 371, whichclaims priority to Japanese Patent Application Nos. 2013-214206 (filedon Oct. 11, 2013), 2013-153479 (filed on Jul. 24, 2013), and 2013-147088(filed on Jul. 12, 2013), which are all hereby incorporated by referencein their entirety.

TECHNICAL FIELD

The present disclosure relates to an image decoding device and method,and particularly, relates to an image decoding device capable ofrecognizing performance necessary for decoding more accurately and amethod.

BACKGROUND ART

In recent years, in order to further improve coding efficiency overMPEG-4 Part10 (Advanced Video Coding, hereinafter referred to as “AVC”),Joint Collaboration Team-Video Coding (JCTVC), which is a jointstandardization organization of International Telecommunication UnionTelecommunication Standardization Sector (ITU-T) and InternationalOrganization for Standardization/International ElectrotechnicalCommission (ISO/IEC), has proceeded with standardization of a codingscheme called High Efficiency Video Coding (HEVC) (for example, refer toNon-Patent Literature 1).

In HEVC, it is possible to decode only a region whose decoding isnecessary by an application using a tile structure. In order to indicatethe fact that a tile region is independently decodable, second and laterversions (including MV-HEVC, SHVC, Range Ext. and the like) of HEVC aresupported by motion-constrained tile sets SEI.

CITATION LIST Patent Literature Non-Patent Literature

[Non-Patent Literature 1] Benjamin Bross, Woo-Jin Han, Jens-Rainer Ohm,Gary J. Sullivan, Ye-Kui Wang, Thomas Wiegand, “High Efficiency VideoCoding (HEVC) text specification draft 10 (for FDIS & Last Call)”,JCTVC-L1003_v34, Joint Collaborative Team on Video Coding (JCT-VC) ofITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 12th Meeting: Geneva, CH,14-23 Jan. 2013.

SUMMARY OF INVENTION Technical Problem

However, as information on a level, which serves as a reference fordetermining whether a decoder can decode a stream, and a buffercapacity, only a value of the entire stream or a value of a layer unitis defined.

Therefore, even in an application that decodes only a part of an entireimage, determination of whether decoding is possible is performed byassuming a load when an entire screen is decoded. Accordingly, there isconcern of an unnecessarily high level decoder being necessary. Inaddition, there is concern of applications to be delivered beingunnecessarily limited accordingly.

The present disclosure has been made in view of the above-mentionedproblems and can recognize performance necessary for decoding moreaccurately.

Solution to Problem

An aspect of the present technology is an image decoding deviceincluding: an acquisition unit configured to acquire coded data of imagedata and decoding load definition information for defining a magnitudeof a load of a decoding process of a partial region of an image of theimage data; a control unit configured to control decoding of the codeddata acquired by the acquisition unit based on the decoding loaddefinition information acquired by the acquisition unit; and a decodingunit configured to decode the coded data acquired by the acquisitionunit under control of the control unit.

The partial region may be independently decodable.

The decoding load definition information may include information fordefining a magnitude of a load of a decoding process of the partialregion according to a level indicating a magnitude of a load of thedecoding process.

The decoding load definition information may include information fordefining a magnitude of a load of a decoding process of the partialregion according to information indicating a size of the partial region.

The decoding load definition information may include information fordefining a magnitude of a load of a decoding process of the partialregion according to information indicating a length in a verticaldirection and information indicating a length in a horizontal directionof the partial region.

The decoding load definition information may be included in supplementalenhancement information (SEI) of an independently decodable partialregion.

The image data may include a plurality of layers, and the decoding loaddefinition information of the plurality of layers may be included in theSEI.

The decoding load definition information may include informationindicating a size of the partial region serving as a reference, and alevel indicating a magnitude of a load of a decoding process of thepartial region.

The partial region may be a tile.

The partial region may be a set of a plurality of tiles.

The decoding load definition information may include information fordefining a maximum magnitude of a load of a decoding process among aplurality of partial regions included in a picture of the image dataaccording to a level indicating a magnitude of a load of the decodingprocess.

The decoding load definition information may include information fordefining a magnitude of a load common in a plurality of partial regionsincluded in a picture of the image data according to a level indicatinga magnitude of a load of the decoding process.

When the plurality of partial regions included in the picture have an Lshape, a magnitude of the load may be defined for a rectangular regionincluding the L shape.

The acquisition unit may further acquire information indicating whetherthe decoding load definition information is set, and when the acquiredinformation indicates that the decoding load definition information isset, acquires the the decoding load definition information.

An aspect of the present technology is an image decoding methodincluding: acquiring coded data of image data and decoding loaddefinition information for defining a magnitude of a load of a decodingprocess of a partial region of an image of the image data; controllingdecoding of the acquired coded data based on the acquired decoding loaddefinition information; and decoding the acquired coded data accordingto the controlling.

In an aspect of the present technology, coded data of image data anddecoding load definition information for defining a magnitude of a loadof a decoding process of a partial region of an image of the image dataare acquired; decoding of the acquired coded data is controlled based onthe acquired decoding load definition information; and the acquiredcoded data is decoded according to the controlling.

Advantageous Effects of Invention

According to the present disclosure, it is possible to code and decodean image. In particular, it is possible to recognize performancenecessary for decoding more accurately.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram describing an exemplary configuration of a codingunit.

FIG. 2 is a diagram illustrating an example of a layered image encodingscheme.

FIG. 3 is a diagram for describing an example of spatial scalablecoding.

FIG. 4 is a diagram for describing an example of temporal scalablecoding.

FIG. 5 is a diagram for describing an example of scalable coding of asignal-to-noise ratio.

FIG. 6 is a diagram describing an exemplary application that performspartial display.

FIG. 7 is a diagram describing another exemplary application thatperforms partial display.

FIG. 8 is a diagram describing an exemplary method of defining adecoding load to which the present technology is applied.

FIG. 9 is a diagram illustrating an extension example of MCTS SEI.

FIG. 10 is a diagram describing an overview of MCTS SEI.

FIG. 11 is a diagram describing an overview of MCTS SEI.

FIG. 12 is a diagram describing an overview of MCTS SEI.

FIG. 13 is a diagram describing an overview of MCTS SEI.

FIG. 14 is a diagram describing an overview of MCTS SEI.

FIG. 15 is a diagram describing an overview of MCTS SEI.

FIG. 16 is a diagram describing an overview of MCTS SEI.

FIG. 17 is a diagram describing an overview of MCTS SEI.

FIG. 18 is a diagram illustrating exemplary transmission of a syntax foreach ROI.

FIG. 19 is a diagram illustrating an extension example of MCTS SEI.

FIG. 20 is a diagram illustrating an exemplary syntax of MCTS SEI.

FIG. 21 is a diagram illustrating an extension example of MCTS SEI.

FIG. 22 is a diagram describing a state of parameter mapping.

FIG. 23 is a diagram describing syntax elements.

FIG. 24 is a diagram illustrating an extension example of MCTS SEI.

FIG. 25 is a diagram describing a state of parameter mapping.

FIG. 26 is a diagram describing syntax elements.

FIG. 27 is a block diagram illustrating a main configuration example ofan image coding device.

FIG. 28 is a block diagram illustrating a main configuration example ofa base layer image coding unit.

FIG. 29 is a block diagram illustrating a main configuration example ofan enhancement layer image coding unit.

FIG. 30 is a block diagram illustrating a main configuration example ofa header information generating unit.

FIG. 31 is a flowchart describing an exemplary flow of image codingprocesses.

FIG. 32 is a flowchart describing an exemplary flow of base layer codingprocesses.

FIG. 33 is a flowchart describing an exemplary flow of enhancement layercoding processes.

FIG. 34 is a flowchart describing an exemplary flow of headerinformation generating processes.

FIG. 35 is a block diagram illustrating a main configuration example ofan image decoding device.

FIG. 36 is a block diagram illustrating a main configuration example ofa base layer image decoding unit.

FIG. 37 is a block diagram illustrating a main configuration example ofan enhancement layer image decoding unit.

FIG. 38 is a block diagram illustrating an exemplary configuration of aheader information analyzing unit.

FIG. 39 is a flowchart describing an exemplary flow of image decodingprocesses.

FIG. 40 is a flowchart describing an exemplary flow of headerinformation analyzing processes.

FIG. 41 is a flowchart describing an exemplary flow of base layerdecoding processes.

FIG. 42 is a flowchart describing an exemplary flow of enhancement layerdecoding processes.

FIG. 43 is a diagram illustrating an example of a multi-view imagecoding scheme.

FIG. 44 is a diagram illustrating a main configuration example of amulti-view image coding device to which the present technology isapplied.

FIG. 45 is a diagram illustrating a main configuration example of amulti-view image decoding device to which the present technology isapplied.

FIG. 46 is a block diagram illustrating an example of a mainconfiguration of a computer.

FIG. 47 is a block diagram illustrating an example of a schematicconfiguration of a television device.

FIG. 48 is a block diagram illustrating an example of a schematicconfiguration of a mobile telephone.

FIG. 49 is a block diagram illustrating an exemplary schematicconfiguration of a recording and reproduction device.

FIG. 50 is a block diagram illustrating an exemplary schematicconfiguration of an imaging device.

FIG. 51 is a block diagram illustrating an example of scalable codinguse.

FIG. 52 is a block diagram illustrating another example of scalablecoding use.

FIG. 53 is a block diagram illustrating still another example ofscalable coding use.

FIG. 54 is a block diagram illustrating an example of a schematicconfiguration of a video set.

FIG. 55 is a block diagram illustrating an example of a schematicconfiguration of a video processor.

FIG. 56 is a block diagram illustrating another example of the schematicconfiguration of the video processor.

DESCRIPTION OF EMBODIMENTS

Hereinafter, aspects (hereinafter referred to as “embodiments”) forimplementing the present disclosure will be described. The descriptionswill proceed in the following order.

-   -   1. First embodiment (decoding load definition of partial image)    -   2. Second embodiment (image coding device)    -   3. Third embodiment (image decoding device)    -   4. Fourth embodiment (multi-view image coding and multi-view        image decoding device)    -   5. Fifth embodiment (computer)    -   6. Sixth embodiment (application example)    -   7. Seventh embodiment (application example of scalable coding)    -   8. Eighth embodiment (set, unit, module, and processor)

1. First Embodiment

<Flow of Standardization of Image Coding>

In recent years, devices in which image information is digitallyhandled, and in this case, in order to transmit and accumulateinformation with high efficiency, image information-specific redundancyis used, and an image is compression-coded employing a coding scheme inwhich an orthogonal transform such as a discrete cosine transform andmotion compensation are used for compression have proliferated. As thecoding scheme, Moving Picture Experts Group (MPEG) is exemplified.

In particular, MPEG2 (ISO/IEC 13818-2) is a standard that is defined asa general-purpose image coding scheme, and generally supports both aninterlaced scanning image and a progressive scanning image as well as astandard resolution image and a high-definition image. For example,MPEG2 is currently being widely used for a wide range of applicationsincluding professional applications and consumer applications. When anMPEG2 compression scheme is used, for example, an interlaced scanningimage having a standard resolution of 720×480 pixels may be assigned acode amount (bit rate) of 4 to 8 Mbps. In addition, when the MPEG2compression scheme is used, for example, an interlaced scanning imagehaving a high resolution of 1920×1088 pixels may be assigned a codeamount (bit rate) of 18 to 22 Mbps. Therefore, it is possible toimplement a high compression rate and good image quality.

MPEG2 is mainly designed for high image quality coding suitable forbroadcast, but does not correspond to a lower code amount (bit rate)than that of MPEG1, that is, a coding scheme of a higher compressionrate. With the proliferation of mobile terminals, it is assumed thatneeds for such a coding scheme will increase in the future. Accordingly,MPEG4 coding schemes have been standardized. A standard of image codingschemes was approved as an international standard ISO/IEC14496-2 inDecember 1998.

Further, in recent years, for the initial purpose of image coding fortelevision conferencing, a standard called H.26L (ITU-T (InternationalTelecommunication Union Telecommunication Standardization Sector) Q6/16VCEG (Video Coding Expert Group)) has been standardized. It is knownthat H.26L requests a greater amount of computation for coding anddecoding than coding schemes of the related art such as MPEG2 or MPEG4,but has a higher coding efficiency. In addition, currently, as a part ofMPEG4 activities, based on H.26L, standardization in which functionsthat are not supported in H.26L are also incorporated to implementhigher coding efficiency is being performed as Joint Model ofEnhanced-Compression Video Coding.

As schedules of standardization, H.264 and MPEG-4 Part10 (Advanced VideoCoding, hereinafter referred to as “AVC”) became international standardsin March 2003.

Further, as extensions of H.264/AVC, standardization of Fidelity RangeExtension (FRExt) including coding tools necessary for professional usesuch as RGB, 4:2:2, or 4:4:4, and 8×8 DCT or a quantization matrixdefined in MPEG-2 was completed in February 2005. Therefore, whenH.264/AVC is used, the coding scheme is also able to appropriatelyrepresent film noise included in a movie and is used for a wide range ofapplications such as a Blu-Ray Disc (trademark).

However, in recent years, needs for higher compression rate codingincluding compression of an image of about 4000×2000 pixels, four timesthat of a high definition image, or delivery of a high definition imagein an environment having a limited transmission capacity such as theInternet, are increasing. Therefore, in previously described VCEG underITU-T, study for increasing coding efficiency continues.

Therefore, currently, in order to further increase coding efficiencyover that of AVC, Joint Collaboration Team-Video Coding (JCTVC), whichis a joint standardization organization of ITU-T and InternationalOrganization for Standardization/International ElectrotechnicalCommission (ISO/IEC), proceeding with a standardization of a codingscheme called High Efficiency Video Coding (HEVC). As a standard ofHEVC, a committee draft, which is a draft specification, has been issuedin January 2013 (for example, refer to Non-Patent Literature 1).

<Coding Scheme>

Hereinafter, the present technology will be described with applicationexamples of image coding and decoding of a High Efficiency Video Coding(HEVC) scheme.

<Coding Unit>

In the Advanced Video Coding (AVC) scheme, a layered structure ofmacroblocks and sub-macroblocks is defined. However, a macroblock of16×16 pixels is not optimal for a large image frame provided in the nextgeneration coding scheme Ultra High Definition (UHD, 4000 pixels×2000pixels).

On the other hand, in the HEVC scheme, as illustrated in FIG. 1 , acoding unit (CU) is defined.

The CU is also called a coding tree block (CTB) and is a partial regionof an image of a picture unit, which similarly serves as the macroblockin the AVC scheme. The latter is fixed to a size of 16×16 pixels. On theother hand, the former has a size that is not fixed, but is designatedin image compression information in respective sequences.

For example, in the sequence parameter set (SPS) included in coded datato be output, a maximum size (largest coding unit (LCU)) and a minimumsize (smallest coding unit (SCU)) of the CU are defined.

In each LCU, in a range equal to or greater than a size of the SCU, whensplit-flag=1 is set, the unit may be divided into CUs having a smallersize. In an example of FIG. 1 , the LCU has a size of 128 and a maximumlevel depth of 5. When a value of split_flag is set to “1,” a CU havinga size of 2N×2N is divided into the next lowest level of CUs having asize of N×N.

Further, the CU is divided into a prediction unit (PU) that is a region(a partial region of an image of a picture unit) serving as a processingunit of intra or inter prediction, and is divided into a transform unit(TU)) that is a region (a partial region of an image of a picture unit)serving as a processing unit of an orthogonal transform. Currently, inthe HEVC scheme, it is possible to use 16×16 and 32×32 orthogonaltransform in addition to 4×4 and 8×8.

In a coding scheme in which the CU is defined and various processes areperformed in units of CUs as in the HEVC scheme described above, themacroblock in the AVC scheme may be considered to correspond to the LCUand the block (sub-block) may be considered to correspond to the CU. Inaddition, a motion compensation block in the AVC scheme may beconsidered to correspond to the PU. However, since the CU has a layeredstructure, the LCU of the topmost level has a size that is generally setto be greater than a macroblock of the AVC scheme, for example, 128×128pixels.

Accordingly, hereinafter, the LCU may include the macroblock in the AVCscheme, and the CU may include the block (sub-block) in the AVC scheme.That is, the term “block” used in the following description refers toany partial region in the picture and has a size, a shape, acharacteristic and the like that are not limited. In other words, the“block” includes any region (processing unit), for example, a TU, a PU,an SCU, a CU, an LCU, a sub-block, a macroblock, or a slice. It isneedless to say that a partial region (processing unit) other than theseis included. When there is a need to limit a size, a processing unit orthe like, it will be appropriately described.

In addition, in this specification, a coding tree unit (CM) is a unitincluding a parameter when processing is performed in the coding treeblock (CTB) of the LCU (a maximum number of the CU) and an LCU base(level) thereof. In addition, the coding unit (CU) of the CTU is a unitincluding a parameter when processing is performed in a coding block(CB) and a CU base (level) thereof.

<Mode Selection>

Meanwhile, in the AVC and HEVC coding schemes, in order to achievehigher coding efficiency, it is important to select an appropriateprediction mode.

As an example of such a selection scheme, a method implemented inreference software (disclosed inhttp://iphome.hhi.de/suehring/tml/index.htm) of H.264/MPEG-4 AVC calledJoint Model (JM) may be exemplified.

In JM, it is possible to select a method of determining two modes, ahigh complexity mode and a low complexity mode, to be described below.In both, a cost function value for each prediction mode Mode iscalculated, and a prediction mode minimizing the value is selected as anoptimal mode for the block or the macroblock.

A cost function in the high complexity mode is represented as thefollowing Equation (1).

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack{{{Cost}\mspace{14mu}\left( {{Mode} \in \Omega} \right)} = {D + {\lambda*R}}}} & (1)\end{matrix}$

Here, Ω denotes an entire set of candidate modes for coding the block orthe macroblock, and D denotes difference energy between a decoded imageand an input image when coding is performed in the prediction mode. λdenotes a Lagrange undetermined multiplier provided as a function of aquantization parameter. R denotes a total amount of codes when coding isperformed in the mode including an orthogonal transform coefficient.

That is, when coding is performed in the high complexity mode, in orderto calculate the parameters, D and R, it is necessary to perform aprovisional encoding process once in all candidate modes. Therefore, ahigher amount of computation is necessary.

A cost function in the low complexity mode is represented as thefollowing Equation (2).

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack{{{Cost}\mspace{14mu}\left( {{Mode} \in \Omega} \right)} = {D + {{{QP}\; 2{{Quant}({QP})}} \star {HeaderBit}}}}} & (2)\end{matrix}$

Here, D denotes difference energy between a prediction image and aninput image unlike the high complexity mode. QP2Quant (QP) is providedas a function of a quantization parameter QP. HeaderBit denotes a codeamount of information belonging to a header in a motion vector or a modehaving no orthogonal transform coefficient.

That is, in the low complexity mode, it is necessary to perform aprediction process in respective candidate modes, but is not necessaryfor the decoded image. Therefore, it is not necessary to perform acoding process. For this reason, a lower amount of computation than thatin the high complexity mode may be implemented.

<Layered Coding>

Incidentally, the image coding schemes such as MPEG2 and AVC describedabove have a scalability function. Scalable coding (layered coding)refers to a scheme in which an image is divided into a plurality oflayers (layered), and coding is performed for each layer. FIG. 2 is adiagram illustrating an example of a layered image coding scheme.

As illustrated in FIG. 2 , in layering of the image, using apredetermined parameter having a scalability function as a reference,one image is divided into a plurality of levels (layers). That is, thelayered image (hierarchical image) includes an image of a plurality oflayers whose predetermined parameter values are different from eachother. The plurality of layers of the layered image include a base layerin which coding and decoding are performed using only an image of itsown layer without using an image of another layer, and a non-base layer(also referred to as an “enhancement layer”) in which coding anddecoding are performed using an image of another layer. The non-baselayer may use an image of the base layer or use an image of anothernon-base layer.

In general, the non-base layer includes data (difference data) of adifference image between its own image and an image of another layer sothat redundancy is reduced. For example, when one image is divided intotwo layers, the base layer and the non-base layer (also referred to asan “enhancement layer”), an image having lower quality than an originalimage may be obtained when only data of the base layer is used.Therefore, when data of the base layer and data of the non-base layerare synthesized, the original image (that is, a high quality image) maybe obtained.

When the image is layered in this manner, it is possible to easilyobtain an image of a variety of levels of quality according tocircumstances. For example, in terminals having a low processingcapacity such as a mobile telephone, image compression information ofonly the base layer is transmitted, and a moving image having a lowspatial and temporal resolution or having low image quality is playedand in terminals having a high processing capacity such as a televisionor a personal computer, image compression information of the enhancementlayer in addition to the base layer is transmitted, and a moving imagehaving a high spatial and temporal resolution or high image quality isplayed, so that a transcoding process is not performed and imagecompression information may be transmitted from a server according to acapability of a terminal or a network.

<Scalable Parameter>

In such layered image coding and layered image decoding (scalable codingand scalable decoding), a parameter having a scalability function isarbitrary. For example, a spatial resolution illustrated in FIG. 3 maybe used as a parameter thereof (spatial scalability). In the spatialscalability, a resolution of an image is different for each layer. Thatis, as illustrated in FIG. 3 , each picture is divided into two layers,and the base layer having a lower spatial resolution than the originalimage and the enhancement layer in which synthesizing with the image ofthe base layer is performed and the original image (original spatialresolution) may be obtained. It is needless to say that the number oflevels is only an example, and the image may be layered into any numberof levels.

In addition, as a parameter enabling such a scalable property, anotherexample, for example, a temporal resolution may also be applied(temporal scalability) as illustrated in FIG. 4 . In the temporalscalability, a frame rate is different for each layer. That is, in thiscase, as illustrated in FIG. 4 , the image is divided into layers havingdifferent frame rates, and when a layer having a high frame rate isadded to a layer having a low frame rate, a moving image having a higherframe rate may be obtained, and when all layers are added, an originalmoving image (original frame rate) may be obtained. The number of levelsis only an example, and the image may be layered into any number oflevels.

Further, as a parameter enabling such a scalable property, in anotherexample, for example, as illustrated in FIG. 5 , a signal to noise ratio(SNR) may be applied (SNR scalability). In the SNR scalability, the SNRis different for each layer. That is, in this case, as illustrated inFIG. 5 , each picture is layered into two levels, a base layer having alower SNR than the original image and an enhancement layer that may besynthesized with an image of the base layer to obtain the original image(original SNR). That is, in base layer image compression information,information on an image of a low PSNR is transmitted, and enhancementlayer image compression information is added thereto. Therefore, it ispossible to reconstruct a high PSNR image. It is needless to say thatthe number of levels is only an example and the image may be layeredinto any number of levels.

It is needless to say that the parameter enabling such a scalableproperty may be a parameter other than the above-described example. Forexample, in bit-depth scalability, the base layer is an image of 8 bits,the enhancement layer is added thereto, and thus an image of 10 bits isobtained.

In addition, in chroma scalability, the base layer is a component imageof a 4:2:0 format, the enhancement layer is added thereto, and thus acomponent image of a 4:2:2 format is obtained.

<Definition of Tile Structure and Layer>

In HEVC, it is possible to decode only a region whose decoding isnecessary by an application using a tile structure. In order to indicatethe fact that a tile region is independently decodable, second and laterversions (including MV-HEVC, SHVC, Range Ext. and the like) of HEVC aresupported by motion-constrained tile sets SEI.

Application Examples

Application examples to which the present technology is applied will bedescribed.

In a system configured to deliver an image from a server to a terminal,for example, as exemplified in FIG. 6 , there is an application in whicha single screen is divided into a plurality of screens and delivery isperformed while a display region is switched. In addition, for example,as exemplified in FIG. 7 , there is an application in which a partialregion to be displayed (delivered) is selected in order to select anaspect ratio or a resolution of an image.

In the application of FIG. 6 , partial images are segmented from anentire image using a tile as a unit in coding and decoding of the image,and delivered to terminals. Positions of the segmented partial images inthe entire image may be designated by, for example, a user of theterminal. Therefore, in the terminal, it is possible to display apartial image of a desired position of the entire image. For example, ina service such as sports broadcast, in a wide angle image that isprovided from a server or the like and obtained by capturing an image ofan entire venue, an entire field or the like, focusing on a desired part(for example, a favorite player, a coach, in front of a goal, a bench,and an audience seat) of the user, the partial image may be segmentedand downloaded (or streamed), and displayed on the terminal. That is,the user of the terminal can focus on the desired part of the entireimage.

In the application of FIG. 7 , by simply selecting a tile, a resolutionof a display image may be set to HD or a cinema size.

However, as information on a level, which serves as a reference fordetermining whether a decoder can decode a stream, and a buffercapacity, only a value of the entire stream or a value of a layer unitis defined.

Therefore, even in an application that decodes only a part of an entireimage, determination of whether decoding is possible is performed byassuming a load when an entire screen is decoded. Accordingly, there isconcern of an unnecessarily high level decoder being necessary. Inaddition, there is concern of applications to be delivered beingunnecessarily limited accordingly.

Therefore, decoding load definition information for defining a magnitudeof a load of a decoding process of an independently decodable partialregion of an image of image data to be coded is set, and the decodingload definition information is transmitted. For example, the decodingload definition information is transmitted from a coding side to adecoding side together with coded data of image data.

In this manner, the decoder may recognize performance necessary fordecoding the partial region according to the decoding load definitioninformation, and determine whether decoding is possible. That is, it ispossible to recognize performance necessary for decoding moreaccurately. Therefore, it is possible to select the decoder havingappropriate performance for the image data. Therefore, it is possible tosuppress a situation of applying the decoder having an unnecessarilyhigh level with respect to a decoding load of image data from occurring.In addition, it is possible to suppress applications to be deliveredfrom being unnecessarily limited accordingly.

<Setting of Decoding Load Definition Information>

The decoding load definition information is defined according to, forexample, FIG. 8 . For example, as exemplified in A of FIG. 8 , thedecoding load definition information for defining a magnitude of a loadof a decoding process of the partial region may be set for anindependently decodable partial region of a single layer. In the exampleof A of FIG. 8 , level 4.0 is set for decoding an entire image of asingle layer, and level 2.0 is set for decoding an independentlydecodable tile (partial region) of the image.

In addition, for example, as exemplified in B of FIG. 8 , the decodingload definition information may be set for an independently decodablepartial region of each layer of an image including a plurality oflayers. In the example of B of FIG. 8 , level 5.0 is set for decoding animage of all layers, and level 4.0 is set for decoding an entire imageof the base layer (layer 0). Further, level 2.0 is set for decoding anindependently decodable tile (partial region) of the image of the baselayer (layer 0). Further, level 4.0 is set for decoding (that is,decoding of a tile of the base layer (layer 0) to be referred to and atile of the enhancement layer (layer 1) referring thereto) a tile thatrefers to only an independently decodable tile of an image of the baselayer (layer 0) of an image of an enhancement layer (layer 1).

Further, for example, as exemplified in C of FIG. 8 , the decoding loaddefinition information may be set for the entire image of the layerreferring to only the independently decodable partial region and thereferred partial region. That is, a side that refers to theindependently decodable tile (partial region) may be the entire imagerather than the partial region. In the example of C of FIG. 8 ,basically, the same level as in the example of B of FIG. 8 is set.However, in the example of B of FIG. 8 , the level is set for decoding atile referring to only an independently decodable tile of the image ofthe base layer (layer 0) in the image of the enhancement layer (layer1). However, in the example of C of FIG. 8 , alternatively, level 4.0 isset for decoding (that is, decoding the tile of the base layer (layer 0)to be referred to and the entire image of the enhancement layer(layer 1) referring thereto) the entire image of the enhancement layer(layer 1) referring to only the independently decodable tile of theimage of the base layer (layer 0).

Also, in this case, in order to identify a partial region (tile) (aposition thereof) referenced by the entire image of the enhancementlayer (layer 1), position information of the tile of the base layer(layer 0) serving as a reference source may be associated with (mappedwith) the entire image of the enhancement layer (layer 1) serving as areference source. In the example of C of FIG. 8 , coordinates of thesame position in the tile of the base layer (layer 0) are mapped withupper-left corner coordinates in the entire image of the enhancementlayer (layer 1).

<Parameter Defined by Level>

Note that, a parameter defined by a level includes a maximum pixelnumber (MaxLumaPs), a maximum buffer capacity (MaxCPB Size), a maximumnumber of pixels (MaxLumaSr) of an image per second, a maximum bit rate(MaxBR) of an image or the like.

<Decoding Load Definition Information>

Definition of a magnitude of a load necessary for decoding is performedby extending, for example, motion constrained tile set supplementalenhancement information (MCTS SEI).

For example, as in a described syntax A of FIG. 9 , in MCTS SEI, as thedecoding load definition information for defining a magnitude of a loadof a decoding process of the independently decodable partial region, alevel (mcts_level_idc[i]) indicating a magnitude of a load of a decodingprocess of the partial region may be set. Here, “i” denotes a set (alsoreferred to as a “tile set”), which is the partial region composed of asingle tile or a plurality of tiles. That is, in the example of A ofFIG. 9 , a value of level information (mcts_level_idc) necessary fordecoding is set for each set. In this case, semantics may be asdescribed in, for example, B of FIG. 9 .

In MCTS SEI, the independently decodable partial region is set for eachrectangular set. For example, when an upper-left shaded part of A ofFIG. 10 is the independently decodable partial region, the partialregion is set for each set, in MCTS SEI, as illustrated in B of FIG. 10. Also, as exemplified in C of FIG. 10 , a tile included in the set mayoverlap another set. The number of pixels of the partial region may becalculated from the number of pixels of each set, for example, asexemplified in D of FIG. 10 .

In addition, for example, as in a described syntax A of FIG. 11 , inMCTS SEI, as the decoding load definition information for defining amagnitude of a load of a decoding process of the independently decodablepartial region of a plurality of layers, a level (mcts_level_idc[i][j])indicating a magnitude of a load of a decoding process of the partialregion of each layer may be set. Here, “i” denotes a set and “j” denotesa layer. That is, in the example of A of FIG. 11 , a value of levelinformation (mcts_level_idc) necessary for decoding is set for each setand for each layer. In this case, semantics may be as described in, forexample, B of FIG. 11 .

Further, for example, as in a described syntax A of FIG. 12 , in MCTSSEI, as the decoding load definition information for defining amagnitude of a load of a decoding process of the independently decodablepartial region, information (maxLumaP S_in_set[i]) indicating a size ofthe partial region may be set. Here, “i” denotes a set. That is, in theexample of A of FIG. 12 , a value of information (maxLumaP S_in_set)indicating a size of the set (partial region) is set for each set. Inthis case, semantics may be as described in, for example, B of FIG. 12 .

In addition, for example, as in a described syntax A of FIG. 13 , inMCTS SEI, as the decoding load definition information for defining amagnitude of a load of a decoding process of the independently decodablepartial region, information (mcts_height_in_luma_samples[i]) indicatinga length in a vertical direction and information(mcts_width_in_luma_samples[i]) indicating a length in a horizontaldirection of the partial region may be set. Here, “i” denotes a set.That is, in the example of A of FIG. 13 , a value of information(mcts_height_in_luma_samples) indicating a length in a verticaldirection and a value of information (mcts_width_in_luma_samples)indicating a length in a horizontal direction of the set (partialregion) are set for each set. In this case, semantics may be asdescribed in, for example, B of FIG. 13 .

Further, for example, as in a syntax described in FIG. 14 , in MCTS SEI,as the decoding load definition information for defining a magnitude ofa load of a decoding process of the independently decodable partialregion, a parameter (mcts_hrd_parameters ( )) of a virtual referencedecoder configured to decode the partial region may be set.

In this case, for example, as in a described syntax A of FIG. 15 , asthe parameter (mcts_hrd_parameters ( )) of the virtual referencedecoder, a maximum input bit rate (mcts_bit_rate_value_minus1) and abuffer capacity (mcts_cpb_size_value_minus1) of the virtual referencedecoder may be set. In this case, semantics may be as described in, forexample, B of FIG. 15 .

In addition, for example, as illustrated in A of FIG. 16 , in additionto the extension of MCTS SEI described above, in the sequence parameterset (SPS), information (mcts_present_flag) indicating whether thedecoding load definition information described above is set in MCTS SEImay be set. In this case, semantics may be as described in, for example,B of FIG. 16 .

Further, for example, as illustrated in A of FIG. 17 , in the sequenceparameter set (SPS), instead of the information indicating whether thedecoding load definition information is set, the same decoding loaddefinition information as the decoding load definition information setin MCTS SEI may be set. In this case, semantics may be as described in,for example, B of FIG. 17 .

Also, information set in the sequence parameter set (SPS) may be set ina video parameter set (VPS), instead of the sequence parameter set(SPS).

It is needless to say that a method of setting the decoding loaddefinition information is arbitrary, and is not limited to theabove-described example. In addition, the above-described plurality ofmethods may be combined. Further, the above-described method may becombined with other methods.

As described above, when the decoding load definition information fordefining a magnitude of a load of a decoding process of theindependently decodable partial region is set, it is possible torecognize performance necessary for decoding more accurately based onthe decoding load definition information. In addition, when the decodingload definition information is transmitted to the decoding side, it ispossible to recognize performance necessary for decoding more accuratelyeven on the decoding side.

<Adaptation to DASH>

For example, in a use case of DASH as illustrated in FIG. 6 , it ispreferable that a certain number of tile regions be moved andreproduced. However, when all partial images (a combination of tiles) tobe reproduced are registered as a tile set (tile set), and the decodingload definition information is set therefor, there is a possibility ofan amount of information being increased when the number of tile sets isgreat.

For example, as exemplified in FIG. 18 , when a partial image (tile set)of 2×2 tiles is reproduced (displayed), if a position of the partialimage is moved (when a part to be displayed in the entire image ischanged), all partial images (tile sets) during a movement thereof aredisplayed (reproduced). The number of such partial images may becomeenormous, as shown in the equation on the right side of the drawing.Accordingly, when the decoding load definition information is set forall such partial images, there is a possibility of an amount ofinformation being impractically increased. In addition, there is apossibility of redundancy of decoding load information of each tile sein this case becoming extremely high. That is, there is a possibility ofan amount of unnecessary information being increased.

Here, for example, MCTS SEI is extended, and as the decoding loadinformation, information indicating a size of a partial image serving asa reference and a level indicating a magnitude of a load of a decodingprocess of the partial image are set. That is, information from which itis possible to estimate a magnitude of a load of a decoding processaccording to the size of the partial image is stored in MCTS SEI.

Therefore, for example, when such information is referred to (when asize of a region to be decoded is compared with a size of the partialimage serving as a reference thereof), it is possible recognize to amagnitude of a load of a decoding process of the region more accurately.

Here, the size of the partial image serving as the reference may beindicated by any information, and may be indicated, for example, inunits of tiles obtained by uniformly dividing the entire image, inaddition, the number of sizes serving as a reference thereof may bearbitrary, but it is preferable that the number of sizes be plural inorder for a magnitude of a load of a decoding process to be recognizedmore accurately. In addition, a magnitude of a load of a decodingprocess may be indicated by any information, and may be indicated by,for example, level information (level).

Incidentally, in an existing ROI, it is assumed that an application candesignate any position, or in other words, it is absolutely necessarythat some region (ROI) be defined.

However, an application such as tiled streaming of DASH has a concept ofsegmenting and displaying a region selected by the user (also includingswitching to a stream having a different resolution). That is, since theuser can arbitrarily determine a region to be selected, it is assumedthat all tiles are independently decodable tiles and are furtheruniformly divided and the number of tiles to be selected differingaccording to a capability (level) of a device to be reproduced isassumed to correspond to operation of the most common service.

Therefore, assumption of an application for the existing ROI andassumption of an application for tiled streaming of DASH have slightlydifferent directions.

Accordingly, a mode of an application is introduced such that a level ofa region (tile) unit decoded by the application may be defined whileboth of the assumptions are satisfied in one SEI, and information to bedefined may be changed for each mode for extension.

For example, a concept of a mode is defined for each application tochange information to be defined as necessary. For example, a DASH modeis set. Therefore, it is defined that the DASH mode== “uniform divisionand independence of all tiles.” That is, in the DASH mode, the screen isassumed to be uniformly divided into tiles (uniform_spacing_flag=1@PPS).In addition, individual tiles are assumed to be independently decodable.

Therefore, in the DASH mode, the number of tiles to be decoded at a timeand level information corresponding thereto are described (defined). Forexample, in FIG. 19 , as the number of tiles to be decoded at a time,three cases, 4, 12, and 30, are set. Level information (decoding loadinformation) is set for each of the three cases.

In this manner, it is possible to set the decoding load information moreappropriately for both of the applications described above.

FIG. 20 illustrates an extension example (an exemplary syntax) of MCTSSEI in this case. In the example of FIG. 20 , a mode (mcts_mode) of anapplication is set in a 2nd row from the top. Therefore, as described ina 4th row from the top, when the mode is a mode (mode for current ROIapplication) of an application for the existing ROI (mcts_mode==0), thedecoding load information is set for each independently decodablepartial image, similarly to each example described above. For example,level information (mcts_level_idc[i]) necessary for decoding is set in a13th row from the top.

In addition, when the mode of the application is the DASH mode (mode forDASH application) (mcts_mode==1), the number of tiles of a region to bedecoded at a time and level information corresponding thereto are set.For example, in the 17th row from the top, identification information(mcts_id[i]) of the region is set. In the next row, information(num_of_tiles_minus1[i]) indicating the number of tiles included in theregion indicated by the identification information is set. Further, inthe next row, level information (mcts_level_idc[i]) necessary fordecoding the region is set.

Here, “i” denotes a set. That is, in the example of FIG. 20 , a value oflevel information (mcts_level_idc) necessary for decoding, a value ofidentification information (mcts_id), and a value of information(num_of_tiles_minus1) indicating the number of tiles included in aregion indicated by the identification information are set for each set.

When the decoding load information is set as described above, it ispossible to recognize performance necessary for decoding more accuratelyaccording to a size (the number of tiles) of the region to be decodedbased on the decoding load information. In addition, when the decodingload definition information is transmitted to the decoding side, it ispossible to recognize performance necessary for decoding more accuratelyaccording to a size (the number of tiles) of the region to be decoded,even on the decoding side.

<Other Example 1 of Setting Decoding Load Definition Information>

A rectangular tile set has been described above. In this case, forexample, when the MCTS (independently decodable tile group (partialregion)) is assumed to have an “L” shape, it is necessary to define twosets, a tile set in which a vertical direction indicating a verticalline part of the letter “L” is defined as a longitudinal direction, anda tile set in which a horizontal direction indicating a horizontal linepart of the letter “L” is defined as a longitudinal direction.

In addition, it has been described above that, as the decoding loaddefinition information, a value of information (mcts_level_idc[i])indicating a level (level necessary for decoding) indicating a magnitudeof a load of a decoding process of the partial region is set for eachrectangular set. That is, in this case, in the L-shaped partial region,it is necessary to set two pieces of information (mcts_level_idc[i])indicating a level necessary for decoding. It is needless to say that itis possible to deal with a case in which levels necessary for decodingsets are different in this manner. However, when levels necessary fordecoding sets are the same, it becomes redundant and there is apossibility of coding efficiency decreasing.

Therefore, one level may be set for the independently decodable partialregion rather than for each set. An independently decodable set may bethe independently decodable partial region. That is, a common level maybe defined for a plurality of independently decodable partial regions.An exemplary syntax in this case is described in A of FIG. 21 . Inaddition, an example of semantics in this case is described in B of FIG.21 .

In the example of A of FIG. 21 , in a 3rd row from the top, information(each_tile_one_tile_set_flag) indicating whether all tiles form anindependently decodable set is set. In a 4th row from the top,information (mcts_level_idc_present_flag) indicating whether information(mcts_level_idc) indicating a level necessary for decoding is includedis set in motion-constrained tile sets defined in an SEI message.

Therefore, in a 5th row from the top, when it is determined that alltiles do not form a uniquely decodable set(!each_tile_one_tile_set_flag), in a loop of a 7th row to a 16th rowfrom the top, settings are performed for each set, and a level(mcts_level_idc) necessary for decoding is set in an 18th row from thetop other than the loop.

That is, as exemplified in FIG. 22 , when all tiles are independentlydecodable, a set in which “0”th identification information (mcts_id[0])is assigned and a set in which “1”st identification information(mcts_id[1]) is assigned are adjacent to each other, and levelsnecessary for decoding sets are the same, level information(mcts_level_idc[i]) is not set for each set, but level information(mcts_level_idc) common to both sets may be set.

In this manner, only one (common) piece of level information may be setfor a plurality of independently decodable partial regions (for example,partial regions necessary to be represented by a plurality of sets).Therefore, it is possible to reduce redundancy and increase codingefficiency.

Also, in HEVC described in Non-Patent Literature 1, it is assumed thatthe level is defined in an entire image (picture) unit, and definitionof a parameter of the level is also performed in the entire image(picture) unit. Therefore, when the level is defined in a partial regionunit described above, definition of the parameter of the level for thepartial region is also performed, and the definition may be assigned(parameter mapping is performed) to definition of the level of theentire image (picture) unit.

For example, when the independently decodable partial region has arectangular shape, the parameter of the level for the partial region isdefined. The definition of the level of the entire image (picture) unitmay be replaced with the definition. In addition, for example, when theindependently decodable partial region has an “L” shape, a rectangleincluding the L-shaped partial region is set, the parameter of the levelfor the rectangular region is defined, and definition of the level ofthe entire image (picture) unit may be replaced with the definition.

For example, in FIG. 22 , when the independently decodable partialregion includes the set of identification information (mcts_id[0]) andthe set of identification information (mcts_id[1]), a rectangular regionincluding both of the sets is set, and is assigned to a picture (a unitof definition of the parameter of the level). That is, a size W of therectangular region in a horizontal direction is estimated aspic_width_in_luma_samples, a size H of the rectangular region in avertical direction is estimated as pic_height_in_luma_samples, a sizeW×H of the rectangular region is estimated as PicSizeInSamplesY, and theparameter of the level for the rectangular region may be defined.

In this manner, a value of the parameter defined for the rectangularregion may be mapped with the parameter defined in the picture unit.Therefore, when such parameter mapping is performed, it is possible toemploy more appropriate definition for the independently decodablepartial region.

Incidentally, when all tiles are independently decodable, it is possibleto set a level (mcts_level_idc[i]) necessary for decoding the partialregion according to a size of the partial region. In this case, the sizeof the partial region may be represented by the number of tiles (thenumber of rows) in a vertical direction and the number of tiles (thenumber of columns) in a horizontal direction of the partial region.Here, “i” denotes the number of correspondence relations between thesize and the level of the partial region.

In the example of A of FIG. 21 , in a loop of a 20th row to a 28th rowfrom the top, the size and the level of the partial region areassociated. In the loop, information (num_mc_tile_columns_minus1[i])indicating the number of tiles of the partial region in a verticaldirection is set (a 24th row from the top), and information(num_mc_tile_rows_minus1[i]) indicating the number of tiles of thepartial region in a horizontal direction is set (a 25th row from thetop).

For example, in FIG. 23 , in a “0”th correspondence relation in whichidentification information (mcts_level_id[0]) is set, the partial regionof 2×2 tiles is associated with a level (meta_level_idc[0]). That is, inthe correspondence relation, a value of information(num_mc_tile_columns_minus1[0]) indicating the number of tiles of thepartial region in a vertical direction is set to “1,” and information(num_mc_tile_rows_minus1[0]) indicating the number of tiles of thepartial region in a horizontal direction is set to “1.” When suchinformation is set, in addition to the fact that the number of tiles ofthe partial region corresponding to the level (meta_level_idc[0]) is 4,a shape (a rectangle of two vertical tiles×two horizontal tiles) of thepartial region is also shown.

For example, in FIG. 23 , in a “1”st correspondence relation in whichidentification information (mcts_level_id[1]) is set, the partial regionof 4×4 tiles is associated with a level (meta_level_idc[1]). That is, inthe correspondence relation, a value of information(num_mc_tile_columns_minus1[1]) indicating the number of tiles of thepartial region in a vertical direction is set to “3,” and information(num_mc_tile_rows_minus1[1]) indicating the number of tiles of thepartial region in a horizontal direction is set to “3.” When suchinformation is set, in addition to the fact that the number of tiles ofthe partial region corresponding to the level (meta_level_idc[1]) is 16,a shape (a rectangle of four vertical tiles×four horizontal tiles) ofthe partial region is also shown.

In this manner, it is possible to increase convenience of informationfor associating the partial region with the level. For example, aterminal configured to acquire and display a partial image thereofdetermines whether the partial region is horizontally long based on theinformation. When the partial region is horizontally long, it ispossible to adjust an aspect ratio of a display image such as display byinserting a black band on the top and bottom more easily.

<Other Example 2 of Setting Decoding Load Definition Information>

When all tiles are independently decodable, a maximum value(max_level_idc) of a level of a tile unit in the picture may be set.That is, one tile is set as one set, a maximum value in the picture ofthe level se for each set may be set. An exemplary syntax in this caseis described in A of FIG. 24 . In addition, an example of semantics inthis case is described in B of FIG. 24 .

In the example of A of FIG. 24 , in a 21st row from the top, a maximumvalue (max_level_idc) of the level in the picture is set. As exemplifiedin FIG. 26 , the level is a level that is set for the set composed ofone tile. For example, in FIG. 26 , a maximum value of the level set foreach set of 60 sets (60 tiles) in total including 6 vertical sets (6tiles)×10 horizontal sets (10 tiles) in the picture is set asmax_level_idc.

The number of tiles segmented from the entire image is determined by theapplication. Although the number of tiles corresponding to each level isnot entirely defined, the application can sufficiently accuratelydetermine the number of tiles that can be segmented (can perform aprocess such as decoding) based on the maximum value (max_level_idc) ofthe level.

That is, instead of entirely defining the number of tiles correspondingto each level, when only a maximum value (max_level_idc) of the levelset for each set (that is, for each tile) composed of one tile is set inthe picture, the application may control the number of segmented tilesbased on the setting so that the process such as decoding does not fail.

Therefore, in this manner, compared to when the number of tilescorresponding to each level is entirely defined, a syntax of all tilescan be simplified and a load of the process can be reduced. In addition,compared to when the number of tiles corresponding to each level isentirely defined, an amount of information to be transmitted can bereduced and coding efficiency can increase.

Also, as exemplified in A of FIG. 24 , in a 17th row from the top, alevel (mcts_level_idc[i]) for each partial region is set. That is, asexemplified in A of FIG. 25 , when there are two independently decodablesets, a set to which identification information (mcts_id[0]) is assignedand a set to which identification information (mcts_id[1]) is assigned,it is possible to set a level (mcts_level_idc[i]) for each set.

In this case, a rectangular region including the set is set for eachset, and may be assigned to each picture (a unit of defining theparameter of the level). For example, in A of FIG. 25 , the rectangularregion including the set of identification information (mcts_id[0]) isset, a size W of the rectangular region in a horizontal direction isestimated as pic_width_in_luma_samples, a size H of the rectangularregion in a vertical direction is estimated aspic_height_in_luma_samples, a size W×H of the rectangular region isestimated as PicSizeInSamplesY, and the parameter of the level for therectangular region may be defined. Similarly, a rectangular regionincluding the set of identification information (mcts_id[1]) is set, asize W of the rectangular region in a horizontal direction is estimatedas pic_width_in_luma_samples, a size H of the rectangular region in avertical direction is estimated as pic_height_in_luma_samples, a sizeW×H of the rectangular region is estimated as PicSizeInSamplesY, and theparameter of the level for the rectangular region may be defined.

In this manner, a value of the parameter defined for each rectangularregion may be mapped with the parameter defined in the picture unit.Therefore, when such parameter mapping is performed, it is possible toemploy more appropriate definition for each independently decodablepartial region.

Also, when the independently decodable partial region is formed of aplurality of rectangular regions, a rectangle including all theserectangular regions may be set. For example, in B of FIG. 25 , a set ofidentification information (mcts_id[0]) includes two rectangularregions, a rectangular region including all of the set is set, a size Wof the rectangular region in a horizontal direction is estimated aspic_width_in_luma_samples, a size H of the rectangular region in avertical direction is estimated as pic_height_in_luma_samples, a sizeW×H of the rectangular region is estimated as PicSizeInSamplesY, and theparameter of the level for the rectangular region may be defined.

In this manner, even when the independently decodable partial region isformed of a plurality of rectangular regions, a value of the parameterdefined for the rectangular region may be mapped with the parameterdefined in the picture unit. Therefore, when such parameter mapping isperformed, it is possible to employ more appropriate definition of thepartial region even if the independently decodable partial region isformed of a plurality of rectangular regions.

2. Second Embodiment

<Image Coding Device>

Next, a device configured to implement the present technology describedabove and a method thereof will be described. FIG. 27 is a diagramillustrating an image coding device, which is an aspect of an imageprocessing device to which the present technology is applied. An imagecoding device 100 illustrated in FIG. 27 is a device configured toperform layered image coding (scalable coding). As illustrated in FIG.27 , the image coding device 100 includes a base layer image coding unit101, an enhancement layer image coding unit 102, a multiplexing unit103, and a control unit 104.

The base layer image coding unit 101 codes a base layer image andgenerates a base layer image coding stream. The enhancement layer imagecoding unit 102 codes an enhancement layer image and generates anenhancement layer image coding stream. The multiplexing unit 103multiplexes the base layer image coding stream generated in the baselayer image coding unit 101 and the enhancement layer image codingstream generated in the enhancement layer image coding unit 102, andgenerates a layered image coding stream. The multiplexing unit 103transmits the generated layered image coding stream to the decodingside.

The control unit 104 performs settings related to all image data,controls the base layer image coding unit 101 and the enhancement layerimage coding unit 102 based on the settings, and thus controls coding ofeach of the layers. In addition, the control unit 104 generates thevideo parameter set (VPS) using the settings, supplies the parameter tothe multiplexing unit 103, and transmits the parameter to the decodingside. In this case, the video parameter set may be transmitted to beincluded in the layered image coding stream or may be transmitted asdata separate from the layered image coding stream.

In addition, when the decoding load definition information orinformation indicating whether the decoding load definition informationis set is set in the video parameter set (VPS), the control unit 104collects the decoding load definition information or the like from thebase layer image coding unit 101 and the enhancement layer image codingunit 102, and sets the decoding load definition information orinformation indicating whether the decoding load definition informationis set in the video parameter set (VPS) based on the information.

In addition, the base layer image coding unit 101 and the enhancementlayer image coding unit 102 may exchange decoding load relatedinformation, which is information on a magnitude of a load of a decodingprocess, with each other. For example, as exemplified in FIG. 11 , whenthe decoding load definition information of a plurality of layers isset, a coding unit of the layer collects decoding load relatedinformation of other layers.

<Base Layer Image Coding Unit>

FIG. 28 is a block diagram illustrating a main configuration example ofthe base layer image coding unit 101 of FIG. 27 . As illustrated in FIG.28 , the base layer image coding unit 101 includes an A/D conversionunit 111, a screen rearrangement buffer 112, a computation unit 113, anorthogonal transform unit 114, a quantization unit 115, a reversiblecoding unit 116, an accumulation buffer 117, an inverse quantizationunit 118, and an inverse orthogonal transform unit 119. In addition, thebase layer image coding unit 101 includes a computation unit 120, a loopfilter 121, a frame memory 122, a selection unit 123, an intraprediction unit 124, an inter prediction unit 125, a prediction imageselection unit 126, and a rate control unit 127.

The A/D conversion unit 111 performs A/D conversion of input image data(base layer image information), and supplies and stores the convertedimage data (digital data) in the screen rearrangement buffer 112. Thescreen rearrangement buffer 112 rearranges images of frames of a storeddisplay order according to an order of frames for coding, depending on agroup of picture (GOP), and supplies the image whose frame order isrearranged to the computation unit 113. In addition, the screenrearrangement buffer 112 supplies the image in which the frame order isrearranged to the intra prediction unit 124 and the inter predictionunit 125.

The computation unit 113 subtracts a prediction image supplied from theintra prediction unit 124 or the inter prediction unit 125 through theprediction image selection unit 126 from the image read from the screenrearrangement buffer 112, and outputs difference information thereof tothe orthogonal transform unit 114. For example, in an image on whichintra coding is performed, the computation unit 113 subtracts aprediction image supplied from the intra prediction unit 124 from theimage read from the screen rearrangement buffer 112. In addition, forexample, in an image on which inter coding is performed, the computationunit 113 subtracts a prediction image supplied from the inter predictionunit 125 from the image read from the screen rearrangement buffer 112.

The orthogonal transform unit 114 performs an orthogonal transform suchas a discrete cosine transform or a Karhunen-Loève transform on thedifference information supplied from the computation unit 113. Theorthogonal transform unit 114 supplies a conversion coefficient thereofto the quantization unit 115.

The quantization unit 115 quantizes the conversion coefficient suppliedfrom the orthogonal transform unit 114. The quantization unit 115 setsthe quantization parameter based on information on a target value of acode amount supplied from the rate control unit 127, and performsquantization thereof. The quantization unit 115 supplies the quantizedconversion coefficient to the reversible coding unit 116.

The reversible coding unit 116 codes the conversion coefficientquantized in the quantization unit 115 using an arbitrary coding scheme.Since coefficient data is quantized under control of the rate controlunit 127, the code amount becomes a target value (or approximates atarget value) set by the rate control unit 127.

In addition, the reversible coding unit 116 acquires informationindicating a mode of intra prediction from the intra prediction unit124, and acquires information indicating a mode of inter prediction ordifferential motion vector information from the inter prediction unit125. Further, the reversible coding unit 116 appropriately generates anetwork abstraction layer (NAL) unit of a base layer including asequence parameter set (SPS), a picture parameter set (PPS) and thelike.

The reversible coding unit 116 codes various pieces of information usingan arbitrary coding scheme and sets the information as a part of codeddata (also referred to as a “coding stream”) (multiplexes). Thereversible coding unit 116 supplies and accumulates the coded dataobtained by coding to the accumulation buffer 117.

Examples of the coding scheme of the reversible coding unit 116 includevariable-length coding and arithmetic coding. Examples of thevariable-length coding include context-adaptive variable length coding(CAVLC) defined in an H.264 scheme or the AVC scheme. Examples of thearithmetic coding include context-adaptive binary arithmetic coding(CABAC).

The accumulation buffer 117 temporarily maintains the coding stream(base layer coding stream) supplied from the reversible coding unit 116.The accumulation buffer 117 outputs the maintained base layer codingstream to the multiplexing unit 103 (FIG. 27 ) at a predeterminedtiming. That is, the accumulation buffer 117 also serves as atransmission unit configured to transmit the base layer coding stream.

In addition, the conversion coefficient quantized in the quantizationunit 115 is also supplied to the inverse quantization unit 118. Theinverse quantization unit 118 performs inverse quantization of thequantized conversion coefficient using a method corresponding toquantization by the quantization unit 115. The inverse quantization unit118 supplies the obtained conversion coefficient to the inverseorthogonal transform unit 119.

The inverse orthogonal transform unit 119 performs an inverse orthogonaltransform of the conversion coefficient supplied from the inversequantization unit 118 using a method corresponding to an orthogonaltransform process by the orthogonal transform unit 114. An inverseorthogonal-transformed output (restored difference information) issupplied to the computation unit 120.

The computation unit 120 adds the prediction image from the intraprediction unit 124 or the inter prediction unit 125 through theprediction image selection unit 126 to the restored differenceinformation, which is an inverse orthogonal transform result suppliedfrom the inverse orthogonal transform unit 119, and obtains a locallydecoded image (decoded image). The decoded image is supplied to the loopfilter 121 or the frame memory 122.

The loop filter 121 includes a deblocking filter, an adaptation loopfilter or the like, and performs an appropriate filter process on areconstructed image supplied from the computation unit 120. For example,the loop filter 121 performs a deblocking filter process on thereconstructed image and thus removes block distortion of thereconstructed image. In addition, for example, the loop filter 121performs a loop filter process on the deblocking filter process result(the reconstructed image whose block distortion is removed) using aWiener filter to improve image quality. The loop filter 121 supplies thefilter process result (hereinafter also referred to as a “decodedimage”) to the frame memory 122.

Also, the loop filter 121 may further perform any other filter processon the reconstructed image. In addition, the loop filter 121 may supplyinformation on a filter coefficient or the like used in the filterprocess to the reversible coding unit 116 as necessary, and code theinformation.

The frame memory 122 stores the supplied decoded image, and supplies thestored decoded image to the selection unit 123 as a reference image at apredetermined timing.

More specifically, the frame memory 122 stores the reconstructed imagesupplied from the computation unit 120 and the decoded image suppliedfrom the loop filter 121. The frame memory 122 supplies the storedreconstructed image to the intra prediction unit 124 through theselection unit 123 at a predetermined timing or based on a request fromthe outside such as the intra prediction unit 124. In addition, theframe memory 122 supplies the stored decoded image to the interprediction unit 125 through the selection unit 123 at a predeterminedtiming or based on a request from the outside such as the interprediction unit 125.

The selection unit 123 selects a supply destination of the referenceimage supplied from the frame memory 122. For example, in intraprediction, the selection unit 123 supplies the reference image (a pixelvalue in a current picture or a base layer decoded image) supplied fromthe frame memory 122 to the intra prediction unit 124. In addition, forexample, in inter prediction, the selection unit 123 supplies thereference image (a decoded image other than the current picture of anenhancement layer or the base layer decoded image) supplied from theframe memory 122 to the inter prediction unit 125.

The intra prediction unit 124 performs a prediction process on a currentpicture, which is an image of frames of a process target, and generatesa prediction image. The intra prediction unit 124 performs theprediction process for each predetermined block (using a block as aprocessing unit). That is, the intra prediction unit 124 generates aprediction image of a current block, which is a process target of thecurrent picture. In this case, the intra prediction unit 124 performs aprediction process (in-screen prediction (also referred to as “intraprediction”)) using the reconstructed image supplied from the framememory 122 through the selection unit 123 as the reference image. Thatis, the intra prediction unit 124 generates a prediction image using apixel value in the periphery of the current block included in thereconstructed image. The peripheral pixel value used in the intraprediction is a pixel value of a pixel, which has been processed, of thecurrent picture. In the intra prediction (that is, in a method ofgenerating a prediction image), a plurality of methods (also referred toas “intra prediction modes”) are prepared in advance as candidates. Theintra prediction unit 124 performs intra prediction in the plurality ofintra prediction modes prepared in advance.

The intra prediction unit 124 generates a prediction image in all of theintra prediction modes serving as candidates, evaluates a cost functionvalue of each prediction image using an input image supplied from thescreen rearrangement buffer 112, and selects an optimal mode. When anoptimal intra prediction mode is selected, the intra prediction unit 124supplies the prediction image generated in the optimal mode to theprediction image selection unit 126.

In addition, as described above, the intra prediction unit 124appropriately supplies intra prediction mode information indicating anemployed intra prediction mode or the like to the reversible coding unit116 and codes it.

The inter prediction unit 125 performs a prediction process on thecurrent picture and generates a prediction image. The inter predictionunit 125 performs the prediction process for each predetermined block(using a block as a processing unit). That is, the inter prediction unit125 generates a prediction image of a current block, which is a processtarget of the current picture. In this case, the inter prediction unit125 performs the prediction process using image data of the input imagesupplied from the screen rearrangement buffer 112 and image data of thedecoded image supplied from the frame memory 122 as the reference image.The decoded image is an image (a picture other than the current picture)of frames that are processed before the current picture. That is, theinter prediction unit 125 performs a prediction process (inter-screenprediction (also referred to as “inter prediction”)) of generating aprediction image using an image of another picture.

The inter prediction is performed by motion prediction and motioncompensation. More specifically, the inter prediction unit 125 uses theinput image and the reference image, performs a motion prediction on thecurrent block, and detects a motion vector. Therefore, the interprediction unit 125 uses the reference image, performs a motioncompensation process according to the detected motion vector, andgenerates a prediction image (inter prediction image information) of thecurrent block. In the inter prediction (that is, in a method ofgenerating a prediction image), a plurality of methods (also referred toas “inter prediction modes) are prepared in advance as candidates. Theinter prediction unit 125 performs such inter prediction in theplurality of inter prediction modes prepared in advance.

The inter prediction unit 125 generates a prediction image in all of theinter prediction modes serving as candidates. The inter prediction unit125 uses the input image supplied from the screen rearrangement buffer112 and information of a generated difference motion vector, evaluates acost function value of each prediction image and selects an optimalmode. When an optimal inter prediction mode is selected, the interprediction unit 125 supplies the prediction image generated in theoptimal mode to the prediction image selection unit 126.

When information indicating an employed inter prediction mode or thecoded data is decoded, the inter prediction unit 125 suppliesinformation necessary for performing a process in the inter predictionmode or the like to the reversible coding unit 116 and codes it. Asnecessary information, for example, information of a generateddifference motion vector or a flag indicating an index of a predictedmotion vector as predicted motion vector information is exemplified.

The prediction image selection unit 126 selects a supply source of theprediction image supplied to the computation unit 113 or the computationunit 120. For example, in intra coding, the prediction image selectionunit 126 selects the intra prediction unit 124 as the supply source ofthe prediction image, and supplies the prediction image supplied fromthe intra prediction unit 124 to the computation unit 113 or thecomputation unit 120. In addition, for example, in inter coding, theprediction image selection unit 126 selects the inter prediction unit125 as the supply source of the prediction image, and supplies theprediction image supplied from the inter prediction unit 125 to thecomputation unit 113 or the computation unit 120.

The rate control unit 127 controls a rate of the quantization operationof the quantization unit 115 based on the code amount of the coded dataaccumulated in the accumulation buffer 117 so that no overflow orunderflow occurs.

In addition, the frame memory 122 supplies the stored base layer decodedimage to the enhancement layer image coding unit 102.

In addition, as illustrated in FIG. 28 , the base layer image codingunit 101 further includes a header information generating unit 128.

The header information generating unit 128 generates header informationsuch as the sequence parameter set (SPS) or MCTS SEI. In this case, asdescribed in the first embodiment, the header information generatingunit 128 performs a process of setting the decoding load definitioninformation for defining a magnitude of a load of a decoding process ofthe independently decodable partial region. For example, the headerinformation generating unit 128 may acquire decoding load relatedinformation of the base layer from the reversible coding unit 116 andgenerate the decoding load definition information of the independentlydecodable partial region of the base layer based on the decoding loadrelated information. In addition, the header information generating unit128 may acquire, for example, decoding load related information of theenhancement layer from the enhancement layer image coding unit 102, andgenerate the decoding load definition information of the independentlydecodable partial region of the enhancement layer based on the decodingload related information.

Further, the header information generating unit 128 may supply thedecoding load definition information or the like to the control unit104, and enable settings for the decoding load definition information tobe performed in the video parameter set.

<Enhancement Layer Image Coding Unit>

FIG. 29 is a block diagram illustrating a main configuration example ofthe enhancement layer image coding unit 102 of FIG. 27 . As illustratedin FIG. 29 , the enhancement layer image coding unit 102 basically hasthe same configuration as the base layer image coding unit 101 of FIG.28 .

That is, as illustrated in FIG. 29 , the enhancement layer image codingunit 102 includes an A/D conversion unit 131, a screen rearrangementbuffer 132, a computation unit 133, an orthogonal transform unit 134, aquantization unit 135, a reversible coding unit 136, an accumulationbuffer 137, an inverse quantization unit 138, and an inverse orthogonaltransform unit 139. In addition, the enhancement layer image coding unit102 includes a computation unit 140, a loop filter 141, a frame memory142, a selection unit 143, an intra prediction unit 144, an interprediction unit 145, a prediction image selection unit 146, and a ratecontrol unit 147.

The A/D conversion unit 131 to the rate control unit 147 correspond toand perform the same processes as the A/D conversion unit 111 to therate control unit 127 of FIG. 28 . However, respective units of theenhancement layer image coding unit 102 perform a process of codingenhancement layer image information rather than the base layer.Therefore, when processes of the A/D conversion unit 131 to the ratecontrol unit 147 are described, the above descriptions of the A/Dconversion unit 111 to the rate control unit 127 of FIG. 28 may beapplied. However, in this case, it is necessary that data to beprocessed be data of the enhancement layer rather than data of the baselayer. In addition, it is necessary to appropriately replace aprocessing unit of an input source or an output destination of data witha corresponding processing unit among the A/D conversion unit 131 to therate control unit 147 and read it.

The enhancement layer image coding unit 102 further includes a headerinformation generating unit 148.

The header information generating unit 148 corresponds to the headerinformation generating unit 128 of FIG. 28 and performs the same processas the header information generating unit 128. However, the headerinformation generating unit 148 performs a process of the enhancementlayer rather tan the base layer.

Alternatively, when the decoding load definition information of theenhancement layer is also created in the header information generatingunit 128 of the base layer, the header information generating unit 148of the enhancement layer may be omitted.

<Header Information Generating Unit>

FIG. 30 is a diagram illustrating an exemplary configuration of functionblocks of the header information generating unit 128 of FIG. 28 . In theheader information generating unit 128, for example, when a program readby a CPU from a ROM or the like is executed using a RAM, theabove-described process is executed and thus various function blocksillustrated in FIG. 30 are implemented.

As illustrated in FIG. 30 , the header information generating unit 128includes a decoding load related information acquisition unit 151, anMCTS SEI generating unit 152, and an SPS generating unit 153.

The decoding load related information acquisition unit 151 acquiresinformation on a load of the decoding process that is used to generatethe decoding load definition information of the independently decodablepartial region. As long as it is used to generate the decoding loaddefinition information, any content of information on a load of thedecoding process may be used.

As described in the first embodiment, the MCTS SEI generating unit 152generates MCTS SEI including the decoding load definition information ofthe independently decodable partial region. That is, the MCTS SEIgenerating unit 152 sets the decoding load definition information of theindependently decodable partial region in MCTS SEI. Content of thedecoding load definition information is arbitrary. For example, in thefirst embodiment, any one or more of various pieces of informationdescribed with reference to FIGS. 9 to 26 may be included in thedecoding load definition information. Also, when a level is defined in apartial region unit, the MCTS SEI generating unit 152 also defines aparameter of the level for the partial region, as described in the firstembodiment. The definition may be assigned (parameter mapping isperformed) to definition of the level of the entire image (picture)unit.

As described in the first embodiment, the SPS generating unit 153generates the sequence parameter set (SPS) including the decoding loaddefinition information of the independently decodable partial region orinformation indicating whether the decoding load definition informationof the independently decodable partial region is set in MCTS SEI (alsoreferred to collectively as “information on definition of a decodingload”). That is, the SPS generating unit 153 sets information ondefinition of a decoding load of the independently decodable partialregion in the sequence parameter set (SPS). Content of the informationon definition of a decoding load is arbitrary. For example, in the firstembodiment, any one or more of various pieces of information describedwith reference to FIGS. 9 to 26 may be included in the decoding loaddefinition information.

Also, as described in the first embodiment, the decoding load definitioninformation of the independently decodable partial region is set only inMCTS SEI, and this information may not be set in the sequence parameterset (SPS). In this case, the SPS generating unit 153 may be omitted.

<Flow of Image Coding Processes>

Next, a flow of processes executed by the image coding device 100described above will be described. First, an exemplary flow of imagecoding processes will be described with reference to a flowchart of FIG.31 .

When the image coding process starts, the control unit 104 of the imagecoding device 100 performs settings of entire scalable coding in StepS101.

In Step S102, the control unit 104 controls respective units of the baselayer image coding unit 101 to the multiplexing unit 103 according tosettings performed in Step S101.

In Step S103, the control unit 104 generates a video parameter set (VPS)by applying the settings performed in Step S101.

In Step S104, the base layer image coding unit 101 codes image data ofthe base layer.

In Step S105, the enhancement layer image coding unit 102 codes imagedata of the enhancement layer.

In Step S106, the multiplexing unit 103 multiplexes the base layer imagecoding stream generated in Step S104 and the enhancement layer imagecoding stream generated in Step S105 (that is, bitstreams of thelayers), and generates the layered image coding stream of one system.Also, the multiplexing unit 103 includes the video parameter set (VPS)generated in Step S103 in the layered image coding stream as necessary.The multiplexing unit 103 outputs the layered image coding stream andtransmits the stream to the decoding side.

When the process of Step S106 ends, the image coding device 100 ends theimage coding process. One picture is processed by such image codingprocesses. Therefore, the image coding device 100 repeatedly executessuch image coding processes for each picture of layered moving imagedata. However, processes that are not necessarily performed for eachpicture, for example, the processes of Steps S101 to S103, areappropriately omitted.

<Flow of Base Layer Coding Processes>

Next, in Step S104 of FIG. 31 , an exemplary flow of base layer codingprocesses executed by the base layer image coding unit 101 will bedescribed with reference to a flowchart of FIG. 32 .

When the base layer coding process starts, the AD conversion unit 111 ofthe base layer image coding unit 101 performs A/D conversion of an imageof frames (picture) of the input moving image in Step S121.

In Step S122, the screen rearrangement buffer 112 stores the image onwhich A/D conversion is performed in Step S121, and performsrearrangement according to an order of coding from a display order ofpictures.

In Step S123, the intra prediction unit 124 performs an intra predictionprocess in an intra prediction mode.

In Step S124, the inter prediction unit 125 performs an inter predictionprocess in which motion prediction or motion compensation is performedin the inter prediction mode.

In Step S125, the prediction image selection unit 126 selects theprediction image based on a cost function value or the like. That is,the prediction image selection unit 126 selects any of the predictionimage generated by intra prediction of Step S123 and the predictionimage generated by inter prediction of Step S124.

In Step S126, the computation unit 113 computes a difference between theinput image whose frame order is rearranged in the process of Step S122and the prediction image selected in the process of Step S125. That is,the computation unit 113 generates image data of a difference imagebetween the input image and the prediction image. The image data of thedifference image obtained in this manner has a smaller amount of datathan original image data. Therefore, compared to when the image isdirectly coded, it is possible to compress an amount of data.

In Step S127, the orthogonal transform unit 114 performs orthogonaltransform of the image data of the difference image generated in theprocess of Step S126.

In Step S128, the quantization unit 115 uses the quantization parametercalculated by the rate control unit 127 and quantizes the orthogonaltransform coefficient obtained in the process of Step S127.

In Step S129, the inverse quantization unit 118 performs inversequantization of the coefficient (also referred to as a “quantizationcoefficient”) generated and quantized in the process of Step S128 usinga characteristic corresponding to a characteristic of the quantizationunit 115.

In Step S130, the inverse orthogonal transform unit 119 performs aninverse orthogonal transform of the orthogonal transform coefficientobtained in the process of Step S129.

In Step S131, the computation unit 120 adds the prediction imageselected in the process of Step S125 to the difference image restored inthe process of Step S130, and thus generates image data of thereconstructed image.

In Step S132, the loop filter 121 performs the loop filter process ofthe image data of the reconstructed image generated in the process ofStep S131. Therefore, block distortion of the reconstructed image or thelike is removed.

In Step S133, the frame memory 122 stores data such as the decoded image(the base layer decoded image) obtained in the process of Step S132 orthe reconstructed image obtained in the process of Step S131.

In Step S134, the reversible coding unit 116 codes the coefficientobtained and quantized in the process of Step S128. That is, reversiblecoding such as variable-length coding or arithmetic coding is performedon data corresponding to the difference image.

In addition, in this case, the reversible coding unit 116 codesinformation on a prediction mode of the prediction image selected in theprocess of Step S125, and adds the difference image to the coded dataobtained by coding. That is, the reversible coding unit 116 also codesoptimal intra prediction mode information supplied from the intraprediction unit 124 or information corresponding to the optimal interprediction mode supplied from the inter prediction unit 125 and adds theresult to the coded data.

In Step S135, the header information generating unit 128 generatesheader information of various null units or the like. The generatedheader information is supplied to the reversible coding unit 116 andadded to the coded data.

In Step S136, the accumulation buffer 117 accumulates the coded data(the base layer image coding stream) obtained in the processes of StepS134 and Step S135. The base layer image coding stream accumulated inthe accumulation buffer 117 is appropriately read, supplied to themultiplexing unit 103, multiplexed with the enhancement layer imagecoding stream, and then is transmitted to the decoding side through atransmission path or a recording medium.

In Step S137, the rate control unit 127 controls a rate of thequantization operation of the quantization unit 115 based on the codeamount (an amount of generated codes) of the coded data accumulated inthe accumulation buffer 117 in the process of Step S136 so that nooverflow or underflow occurs. In addition, the rate control unit 127supplies information on the quantization parameter to the quantizationunit 115.

When the process of Step S137 ends, the base layer coding process ends,and the process returns to FIG. 31 .

<Flow of Enhancement Layer Coding Processes>

Next, in Step S105 of FIG. 31 , an exemplary flow of enhancement layercoding processes executed by the enhancement layer image coding unit 102will be described with reference to a flowchart of FIG. 33 .

Respective processes (Steps S141 to S157) of the enhancement layercoding processes correspond to respective processes (Steps S121 to S137)of the base layer coding processes, and are executed basically in thesame manner as these processes. While the processes of the base layercoding processes are performed on the base layer, respective processes(Steps S141 to S157) of the enhancement layer coding processes areperformed on the enhancement layer.

In addition, when the decoding load definition information of theindependently decodable partial region is set only in the base layer,settings of the decoding load definition information may be omitted inStep S155.

When the process of Step S157 ends, the enhancement layer coding processends, and the process returns to FIG. 31 .

<Flow of Header Generation Processes>

Next, an exemplary flow of header generation processes executed in StepS135 of FIG. 32 will be described with reference to a flowchart of FIG.34 .

When the header generation process starts, the header informationgenerating unit 128 generates various pieces of header information, forexample, a sequence parameter set (SPS), SEI, a picture parameter set(PPS), and a slice header (SliceHeader) in Step S161.

In Step S162, the decoding load related information acquisition unit 151acquires decoding load related information, which is information on aload of a decoding process of the partial region, necessary forgenerating the decoding load definition information of the independentlydecodable partial region. Additionally, when the decoding loaddefinition information of the enhancement layer is set, decoding loadrelated information is also acquired from the enhancement layer. As longas it is used to generate the decoding load definition information, anycontent of the decoding load related information may be used.

In Step S163, the MCTS SEI generating unit 152 sets the decoding loaddefinition information of the partial region in MCTS SEI of theindependently decodable partial region generated in Step S161. Forexample, the MCTS SEI generating unit 152 sets the decoding loaddefinition information, as described with reference to syntaxes of FIGS.9 to 26 in the first embodiment.

For example, as illustrated in FIG. 10 , when the independentlydecodable partial region includes a plurality of sets (a plurality oftiles), the MCTS SEI generating unit 152 uses decoding load relatedinformation of each tile, and sets decoding load related information foreach set. In addition, the MCTS SEI generating unit 152 may use decodingload related information of each tile and set the decoding loaddefinition information of the entire partial region.

Also, when there are a plurality of independently decodable partialregions, the MCTS SEI generating unit 152 may set the decoding loaddefinition information for each partial region. Content of the decodingload definition information is arbitrary. For example, in the firstembodiment, any one or more of various pieces of information describedwith reference to FIGS. 9 to 26 may be included in the decoding loaddefinition information.

In Step S164, the SPS setting unit 153 sets the decoding load definitioninformation of the independently decodable partial region, orinformation (also referred to as “information on definition of adecoding load”) indicating whether the decoding load definitioninformation of the partial region is set in MCTS SEI in the sequenceparameter set (SPS) generated in Step S161. Content of the informationon definition of a decoding load is arbitrary. For example, in the firstembodiment, any one or more of various pieces of information describedwith reference to FIGS. 9 to 26 may be included in the decoding loaddefinition information.

When the process of Step S164 ends, the header information generatingprocess ends and the process returns to FIG. 32 .

Also, as described in the first embodiment, the decoding load definitioninformation of the independently decodable partial region is set only inMCTS SEI, and this information may not be set in the sequence parameterset (SPS). In this case, the process of Step S164 may be omitted.

In addition, in Step S163, when a level is defined in a partial regionunit, the MCTS SEI generating unit 152 also defines a parameter of thelevel for the partial region, as described in the first embodiment. Thedefinition may be assigned to definition of the level of the entireimage (picture) unit (parameter mapping is performed).

The header information set as described above is supplied to thereversible coding unit 116 and included in the coded data.

When the respective processes are executed in this manner, the imagecoding device 100 can recognize performance necessary for decoding moreaccurately.

Also, when the decoding load definition information of the independentlydecodable partial region is set in the enhancement layer, the headerinformation generating unit 148 may be executed as described withreference to the flowchart of FIG. 34 . On the other hand, when nodecoding load definition information is set, only the process of StepS161 of FIG. 34 may be executed.

3. Third Embodiment

<Image Decoding Device>

Next, decoding of the coded data coded as described above will bedescribed. FIG. 35 is a block diagram illustrating a main configurationexample of an image decoding device corresponding to the image codingdevice 100, which is an aspect of an image processing device to whichthe present technology is applied. An image decoding device 200illustrated in FIG. 35 decodes coded data generated by the image codingdevice 100 using a decoding method corresponding to the coding method(that is, hierarchically decodes coded data that is hierarchicallycoded). As illustrated in FIG. 35 , the image decoding device 200includes a demultiplexing unit 201, a base layer image decoding unit202, an enhancement layer image decoding unit 203, and a control unit204.

The demultiplexing unit 201 receives the layered image coding stream inwhich the base layer image coding stream and the enhancement layer imagecoding stream transmitted from the coding side are multiplexed,demultiplexes the received stream, and extracts the base layer imagecoding stream and the enhancement layer image coding stream. The baselayer image decoding unit 202 decodes the base layer image coding streamextracted by the demultiplexing unit 201 and obtains the base layerimage. The enhancement layer image decoding unit 203 decodes theenhancement layer image coding stream extracted by the demultiplexingunit 201 and obtains the enhancement layer image.

The control unit 204 analyzes the video parameter set (VPS) suppliedfrom the demultiplexing unit 201, and controls the base layer imagedecoding unit 202 and the enhancement layer image decoding unit 203based on the information (controls coding of each of the layers).

In addition, the control unit 204 acquires the analysis result of thedecoding load definition information of the header information from thebase layer image decoding unit 202 and the enhancement layer imagedecoding unit 203, and controls operations of respective processingunits of the image decoding device 200 according to the analysis result.

<Base Layer Image Decoding Unit>

FIG. 36 is a block diagram illustrating a main configuration example ofthe base layer image decoding unit 202 of FIG. 35 . As illustrated inFIG. 36 , the base layer image decoding unit 202 includes anaccumulation buffer 211, a reversible decoding unit 212, an inversequantization unit 213, an inverse orthogonal transform unit 214, acomputation unit 215, a loop filter 216, a screen rearrangement buffer217, and a D/A conversion unit 218. In addition, the base layer imagedecoding unit 202 includes a frame memory 219, a selection unit 220, anintra prediction unit 221, an inter prediction unit 222, and aprediction image selection unit 223.

The accumulation buffer 211 also serves as a reception unit configuredto receive the transmitted coded data (the base layer image codingstream supplied from the demultiplexing unit 201). The accumulationbuffer 211 receives and accumulates the transmitted coded data, andsupplies the coded data to the reversible decoding unit 212 at apredetermined timing. Information necessary for decoding such asprediction mode information is added to the coded data.

The reversible decoding unit 212 decodes the information that issupplied by the accumulation buffer 211 and coded by the reversiblecoding unit 116 using a decoding scheme corresponding to the codingscheme. The reversible decoding unit 212 supplies the quantizedcoefficient data of the difference image obtained by decoding to theinverse quantization unit 213.

In addition, the reversible decoding unit 212 determines whether theintra prediction mode or the inter prediction mode is selected as anoptimal prediction mode, and supplies information on the optimalprediction mode in a mode that is determined to have been selectedbetween the intra prediction unit 221 and the inter prediction unit 222.That is, for example, when the intra prediction mode is selected as theoptimal prediction mode on the coding side, the information on theoptimal prediction mode (intra prediction mode information) is suppliedto the intra prediction unit 221. In addition, for example, when theinter prediction mode is selected as the optimal prediction mode on thecoding side, the information on the optimal prediction mode (interprediction mode information) is supplied to the inter prediction unit222.

Further, the reversible decoding unit 212 extracts information necessaryfor inverse quantization, for example, a quantization matrix or aquantization parameter, from the coded data, and supplies theinformation to the inverse quantization unit 213.

The inverse quantization unit 213 performs inverse quantization of thequantized coefficient data obtained by decoding of the reversibledecoding unit 212 using a scheme corresponding to a quantization schemeof the quantization unit 115. Also, the inverse quantization unit 213 isthe same processing unit as the inverse quantization unit 118. Theinverse quantization unit 213 supplies the obtained coefficient data(the orthogonal transform coefficient) to the inverse orthogonaltransform unit 214.

The inverse orthogonal transform unit 214 performs an inverse orthogonaltransform of the orthogonal transform coefficient supplied from theinverse quantization unit 213 using a scheme corresponding to anorthogonal transform scheme of the orthogonal transform unit 114 asnecessary. Also, the inverse orthogonal transform unit 214 is the sameprocessing unit as the inverse orthogonal transform unit 119.

According to the inverse orthogonal transform process, the image data ofthe difference image is restored. The restored image data of thedifference image corresponds to the image data of the difference imagebefore an orthogonal transform is performed on the coding side. In thefollowing, the restored image data of the difference image obtained bythe inverse orthogonal transform process of the inverse orthogonaltransform unit 214 is also referred to as “decoded residual data.” Theinverse orthogonal transform unit 214 supplies the decoded residual datato the computation unit 215. In addition, the image data of theprediction image is supplied from the intra prediction unit 221 or theinter prediction unit 222 to the computation unit 215 through theprediction image selection unit 223.

The computation unit 215 uses the decoded residual data and the imagedata of the prediction image, and obtains the image data of thereconstructed image in which the difference image and the predictionimage are added. The reconstructed image corresponds to the input imagebefore the prediction image is subtracted by the computation unit 113.The computation unit 215 supplies the reconstructed image to the loopfilter 216.

The loop filter 216 appropriately performs the loop filter processincluding the deblocking filter process or an adaptive loop filterprocess on the supplied reconstructed image, and generates the decodedimage. For example, the loop filter 216 performs the deblocking filterprocess on the reconstructed image and thus removes block distortion. Inaddition, for example, the loop filter 216 performs the loop filterprocess on the deblocking filter process result (the reconstructed imagewhose block distortion is removed) using a Wiener filter to improveimage quality.

Also, a type of the filter process performed by the loop filter 216 isarbitrary, and a filter process other than the above-described processmay be performed. In addition, the loop filter 216 may perform thefilter process using the filter coefficient supplied from the imagecoding device. Further, the loop filter 216 may omit such a filterprocess and output input data without the filter process.

The loop filter 216 supplies the decoded image (or the reconstructedimage), which is the filter process result, to the screen rearrangementbuffer 217 and the frame memory 219.

The screen rearrangement buffer 217 rearranges an order of frames of thedecoded image. That is, the screen rearrangement buffer 217 rearrangesthe image of frames rearranged in a coding order by the screenrearrangement buffer 112 according to an original display order. Thatis, the screen rearrangement buffer 217 stores the image data of thedecoded image of frames supplied in the coding order in this order,reads the image data of the decoded image of frames stored in the codingorder, and supplies the result to the D/A conversion unit 218 in thedisplay order. The D/A conversion unit 218 performs D/A conversion ofthe decoded image (digital data) of frames supplied from the screenrearrangement buffer 217, and outputs and displays the result on adisplay (not illustrated) as analog data.

The frame memory 219 stores the supplied decoded image, and supplies thestored decoded image to the intra prediction unit 221 or the interprediction unit 222 through the selection unit 220 as the referenceimage at a predetermined timing or based on a request from the outsidesuch as the intra prediction unit 221 or the inter prediction unit 222.

The intra prediction mode information or the like is appropriatelysupplied to the intra prediction unit 221 from the reversible decodingunit 212. The intra prediction unit 221 performs intra prediction in theintra prediction mode (optimal intra prediction mode) used in the intraprediction unit 124, and generates the prediction image. In this case,the intra prediction unit 221 performs intra prediction using the imagedata of the reconstructed image supplied from the frame memory 219through the selection unit 220. That is, the intra prediction unit 221uses the reconstructed image as the reference image (peripheral pixel).The intra prediction unit 221 supplies the generated prediction image tothe prediction image selection unit 223.

Optimal prediction mode information, motion information or the like isappropriately supplied to the inter prediction unit 222 from thereversible decoding unit 212. The inter prediction unit 222 performsinter prediction using the decoded image (the reference image) acquiredfrom the frame memory 219 in the inter prediction mode (optimal interprediction mode) indicated by the optimal prediction mode informationacquired from the reversible decoding unit 212, and generates theprediction image.

The prediction image selection unit 223 supplies the prediction imagesupplied from the intra prediction unit 221 or the prediction imagesupplied from the inter prediction unit 222 to the computation unit 215.Therefore, in the computation unit 215, the prediction image and thedecoded residual data (difference image information) from the inverseorthogonal transform unit 214 are added to obtain the reconstructedimage.

In addition, the frame memory 219 supplies the stored base layer decodedimage to the enhancement layer image decoding unit 203.

The base layer image decoding unit 202 further includes a headerinformation analyzing unit 224. The header information analyzing unit224 acquires header information extracted from the coding stream by thereversible decoding unit 212, and analyzes the information. For example,the header information analyzing unit 224 analyzes the decoding loaddefinition information included in the header information. The headerinformation analyzing unit 224 supplies information indicating theanalysis result to the control unit 204.

<Enhancement Layer Image Decoding Unit>

FIG. 37 is a block diagram illustrating a main configuration example ofthe enhancement layer image decoding unit 203 of FIG. 35 . Asillustrated in FIG. 37 , the enhancement layer image decoding unit 203includes basically the same configuration as the base layer imagedecoding unit 202 of FIG. 36 .

That is, as illustrated in FIG. 37 , the enhancement layer imagedecoding unit 203 includes an accumulation buffer 231, a reversibledecoding unit 232, an inverse quantization unit 233, an inverseorthogonal transform unit 234, a computation unit 235, a loop filter236, a screen rearrangement buffer 237, and a D/A conversion unit 238.In addition, the enhancement layer image decoding unit 203 includes aframe memory 239, a selection unit 240, an intra prediction unit 241, aninter prediction unit 242, and a prediction image selection unit 243.

The accumulation buffer 231 to the prediction image selection unit 243correspond to and perform the same processes as the accumulation buffer211 to the prediction image selection unit 223 of FIG. 36 . However,respective units of the enhancement layer image decoding unit 203perform a process of coding enhancement layer image information ratherthan the base layer. Therefore, when processes of the accumulationbuffer 231 to the prediction image selection unit 243 are described, theabove descriptions of the accumulation buffer 211 to the predictionimage selection unit 223 of FIG. 36 may be applied. However, in thiscase, it is necessary that data to be processed be data of theenhancement layer rather than data of the base layer. In addition, it isnecessary to appropriately replace a processing unit of an input sourceor an output source of data with a corresponding processing unit of theenhancement layer image decoding unit 203 and read it.

Also, the frame memory 239 acquires the base layer decoded imagesupplied from the base layer image decoding unit 202 and stores theimage as, for example, a long term reference frame. The base layerdecoded image is used as the reference image of, for example, interlayer prediction, in the prediction process by the intra prediction unit241 or the inter prediction unit 242.

The enhancement layer image decoding unit 203 further includes a headerinformation analyzing unit 244.

The reversible decoding unit 232 acquires header information such as thesequence parameter set (SPS) or MCTS SEI from the enhancement layerimage coding stream. There is a possibility of the decoding loaddefinition information of the independently decodable partial regionbeing included in the header information. The reversible decoding unit232 supplies the header information to the header information analyzingunit 244.

The header information analyzing unit 244 analyzes the decoding loaddefinition information of the independently decodable partial regionincluded in the supplied header information, and supplies the analysisresult to the control unit 204.

Also, when the decoding load definition information is set only in thebase layer, the header information analyzing unit 244 of the enhancementlayer may be omitted.

<Header Information Analyzing Unit>

FIG. 38 is a diagram illustrating an exemplary configuration of functionblocks of the header information analyzing unit 224 of FIG. 36 . In theheader information analyzing unit 224, for example, when a program readby a CPU from a ROM or the like is executed using a RAM, theabove-described process is executed and thus various function blocksillustrated in FIG. 38 are implemented.

As illustrated in FIG. 38 , the header information analyzing unit 224includes a header information acquisition unit 251, an SPS analyzingunit 252, an MCTS SEI analyzing unit 253, a level specifying unit 254,and a providing unit 255.

The header information acquisition unit 251 acquires various pieces ofheader information supplied from the reversible decoding unit 212. TheSPS analyzing unit 252 analyzes the sequence parameter set (SPS)acquired as header information by the header information acquisitionunit 251. As described in the first embodiment, the information ondefinition of a decoding load of the independently decodable partialregion is included in the sequence parameter set (SPS). Content of theinformation on definition of a decoding load is arbitrary. For example,in the first embodiment, any one or more of various pieces ofinformation described with reference to FIGS. 9 to 26 may be included inthe decoding load definition information.

The MCTS SEI analyzing unit 253 analyzes MCTS SEI acquired as headerinformation by the header information acquisition unit 251. As describedin the first embodiment, the MCTS SEI includes the decoding loaddefinition information of the independently decodable partial region.Content of the decoding load definition information is arbitrary. Forexample, in the first embodiment, any one or more of various pieces ofinformation described with reference to FIGS. 9 to 26 may be included inthe decoding load definition information.

The level specifying unit 254 specifies a level necessary for decodingthe independently decodable partial region based on the analysis resultof the SPS analyzing unit 252 and MCTS SEI. The providing unit 255provides the level specified by the level specifying unit 254 orinformation on a load of the decoding process corresponding to the levelto the control unit 104.

Also, as described in the first embodiment, when a parameter defined ina predetermined region including the partial region is mapped withdefinition of the level of the entire image (picture) unit, the levelspecifying unit 254 or the providing unit 255 may employ the mappingparameter when the level is specified or the level is interpreted.

Also, as described in the first embodiment, the decoding load definitioninformation of the independently decodable partial region is set only inMCTS SEI, and this information may not be set in the sequence parameterset (SPS). In this case, the SPS analyzing unit 252 may be omitted.

<Flow of Image Decoding Processes>

Next, a flow of processes executed by the image decoding device 200described above will be described. First, an exemplary flow of imagedecoding processes will be described with reference to a flowchart ofFIG. 39 .

When the image decoding process starts, in Step S201, the demultiplexingunit 201 of the image decoding device 200 demultiplexes the layeredimage coding stream transmitted from the coding side for each layer.

In Step S202, the reversible decoding unit 212 extracts the headerinformation including the decoding load definition information from thebase layer image coding stream extracted in the process of Step S201.

Alternatively, for example, when the decoding load definitioninformation is also included in the enhancement layer, the reversibledecoding unit 232 similarly performs the process, and extracts theheader information of the enhancement layer.

In Step S203, the header information analyzing unit 224 analyzes theheader information extracted in Step S202, and specifies the levelnecessary for decoding from the decoding load definition information.

In Step S204, the control unit 204 determines whether the coding streamis decodable based on the analysis result of Step S203. When it isdetermined that the coding stream is decodable, the process advances toStep S205.

In Step S205, the base layer image decoding unit 202 decodes the baselayer image coding stream. In Step S206, the enhancement layer imagedecoding unit 203 decodes the enhancement layer image coding stream.

When the process of Step S206 ends, the image decoding process ends.

On the other hand, in Step S204, when it is determined that the codingstream is not decodable, the process advances to Step S207. In thiscase, in Step S207, the control unit 204 performs an error process,which is a predetermined process when normal decoding cannot beperformed.

The error process may be any process. For example, decoding may beforcibly terminated (including being suspended or paused), or a warningsuch as an image or audio may be provided to the user. In addition, forexample, another coding stream having a lower level may be acquired anddecoding may be restarted. Further, for example, occurrence of disorderin the decoded image may be allowed and the coding stream may beforcibly decoded.

When the process of Step S207 ends, the image decoding process ends.

<Flow of Header Information Analyzing Processes>

Next, an exemplary flow of header information analyzing processesexecuted in Step S203 of FIG. 39 will be described with reference to aflowchart of FIG. 40 .

When the header information analyzing process starts, the SPS analyzingunit 252 determines whether the sequence parameter set (SPS) is referredto in Step S211. When the information on definition of a decoding loadof the independently decodable partial region is included in thesequence parameter set (SPS) acquired as the header information and itis determined that the sequence parameter set (SPS) is referred to, theprocess advances to Step S212.

In Step S212, the SPS analyzing unit 252 analyzes the information ondefinition of a decoding load of the independently decodable partialregion included in the sequence parameter set (SPS). Content of theinformation on definition of a decoding load is arbitrary. For example,in the first embodiment, any one or more of various pieces ofinformation described with reference to FIGS. 9 to 26 may be included inthe decoding load definition information. When the analysis ends, theprocess advances to Step S213. On the other hand, in Step S211, when theinformation on definition of a decoding load of the independentlydecodable partial region is not included in the sequence parameter set(SPS) and it is determined that the sequence parameter set (SPS) is notreferred to, the process advances to Step S213.

In Step S213, the MCTS SEI analyzing unit 253 determines whether MCTSSEI is referred to. When the decoding load definition information of theindependently decodable partial region is set in MCTS SEI acquired asthe header information and it is determined that MCTS SEI is referredto, the process advances to Step S214.

In Step S214, the MCTS SEI analyzing unit 253 analyzes the decoding loaddefinition information of the independently decodable partial regionincluded in MCTS SEI. Content of the decoding load definitioninformation is arbitrary. For example, in the first embodiment, any oneor more of various pieces of information described with reference toFIGS. 9 to 26 may be included in the decoding load definitioninformation. When the analysis ends, the process advances to Step S215.On the other hand, in Step S213, when the decoding load definitioninformation of the independently decodable partial region is not set inMCTS SEI and it is determined that MCTS SEI is not referred to, theprocess advances to Step S215.

In Step S215, the level specifying unit 254 specifies a level necessaryfor decoding based on the analysis result of Steps S212 and S214.

In Step S216, the providing unit 255 provides information indicating thelevel necessary for decoding specified in Step S215 to the control unit104.

When the process of Step S216 ends, the header information analyzingprocess ends and the process returns to FIG. 39 .

Also, as described in the first embodiment, when a parameter defined ina predetermined region including the partial region is mapped withdefinition of the level of the entire image (picture) unit, the levelspecifying unit 254 or the providing unit 255 may employ the mappingparameter in the process of Step S215 or Step S216.

Also, as described in the first embodiment, the decoding load definitioninformation of the independently decodable partial region is set only inMCTS SEI, and this information may not be set in the sequence parameterset (SPS). In this case, the process of Step S211 and Step S212 may beomitted.

<Flow of Base Layer Decoding Processes>

When decoding is possible according to the result of determination ofwhether decoding is possible based on the decoding load definitioninformation described above, the base layer decoding process is executedin Step S205 of FIG. 39 . An exemplary flow of the base layer decodingprocesses will be described with reference to a flowchart of FIG. 41 .

When the base layer decoding process starts, in Step S221, theaccumulation buffer 211 of the base layer image decoding unit 202accumulates the transmitted base layer coding stream. In Step S222, thereversible decoding unit 212 decodes the base layer coding streamsupplied from the accumulation buffer 211. That is, image data such asan I slice, a P slice, and a B slice coded by the reversible coding unit116 is decoded. In this case, various pieces of information other thanthe image data included in the bitstream such as the header informationare also decoded.

In Step S223, the inverse quantization unit 213 performs inversequantization of the quantized coefficient obtained in the process ofStep S222.

In Step S224, the inverse orthogonal transform unit 214 performs aninverse orthogonal transform on the coefficient that isinverse-quantized in Step S223.

In Step S225, the intra prediction unit 221 and the inter predictionunit 222 perform the prediction process and generate the predictionimage. That is, the prediction process is performed in the predictionmode, which is determined in the reversible decoding unit 212, appliedwhen coding is performed. More specifically, for example, if intraprediction is applied when coding is performed, the intra predictionunit 221 generates the prediction image in the intra prediction modethat is set as optimum when coding is performed. In addition, forexample, if inter prediction is applied when coding is performed, theinter prediction unit 222 generates the prediction image in the interprediction mode that is set as optimum when coding is performed.

In Step S226, the computation unit 215 adds the prediction imagegenerated in Step S226 to the difference image obtained by the inverseorthogonal transform in Step S225. Therefore, the image data of thereconstructed image is obtained.

In Step S227, the loop filter 216 appropriately performs the loop filterprocess including the deblocking filter process or the adaptive loopfilter process on the image data of the reconstructed image obtained inthe process of Step S227.

In Step S228, the screen rearrangement buffer 217 rearranges frames ofthe reconstructed image on which the filter process is performed in StepS227. That is, an order of frames rearranged when coding is performed isrearranged to an original display order.

In Step S229, the D/A conversion unit 218 performs D/A conversion of theimage whose frame order is rearranged in Step S228. The image is outputto a display (not illustrated) and the image is displayed.

In Step S230, the frame memory 219 stores data such as the decoded imageobtained in the process of Step S227 or the reconstructed image obtainedin the process of Step S226.

When the process of Step S230 ends, the base layer decoding process endsand the process returns to FIG. 39 .

<Flow of Enhancement Layer Decoding Processes>

Similar to the base layer decoding process, when decoding is possibleaccording to the result of determination of whether decoding is possiblebased on the decoding load definition information described above, theenhancement layer decoding process is executed in Step S206 of FIG. 39 .An exemplary flow of the enhancement layer decoding processes will bedescribed with reference to a flowchart of FIG. 42 .

Respective processes (Steps S241 to S250) of the enhancement layerdecoding processes correspond to respective processes (Steps S221 toS230) of the base layer decoding processes of FIG. 41 , and are executedbasically in the same manner as these processes. While respectiveprocesses (Steps S221 to S230) of the base layer decoding processes areperformed on the base layer, respective processes (Steps S241 to S250)of the enhancement layer decoding processes are performed on theenhancement layer.

When the process of Step S250 ends, the enhancement layer decodingprocess ends and the process returns to FIG. 39 .

When the respective processes are executed in this manner, if the imagedecoding device 200 uses the decoding load definition information of theindependently decodable partial region, it is possible to recognizeperformance necessary for decoding more accurately.

An application range of the present technology includes applications toall image coding devices and image decoding devices capable of codingand decoding partial images.

In addition, the present technology may be applied to an image codingdevice and an image decoding device used to receive image information (abitstream) compressed by motion compensation and an orthogonal transformsuch as a discrete cosine transform, similarly to, for example, MPEG, orH.26x, through satellite broadcast, cable television, the Internet, ornetwork media such as a mobile telephone. In addition, the presenttechnology may be applied to an image coding device and an imagedecoding device used to perform a process in storage media such asoptical and magnetic discs and a flash memory.

4. Fourth Embodiment

<Application to Multi-View Image Encoding and Multi-View Image Decoding>

The above-described series of processes can be applied to multi-viewimage encoding and multi-view image decoding. FIG. 43 illustrates anexample of a multi-view image coding scheme.

As illustrated in FIG. 43 , a multi-view image includes images having aplurality of views. The plurality of views of the multi-view imageinclude a base view for which encoding/decoding is performed using onlythe image of its own view without using information of other views andnon-base views for which encoding/decoding is performed usinginformation of other views. In encoding/decoding of a non-base view, theinformation of the base view may be used, and the information of theother non-base view may be used.

That is, a reference relation between views in multi-view image codingand decoding is similar to a reference relation between layers inlayered image coding and decoding. Therefore, in coding and decoding ofthe multi-view image in FIG. 43 , the above-described method may beapplied. In this manner, similarly to the multi-view image, it ispossible to recognize performance necessary for decoding moreaccurately.

<Multi-View Image Encoding Device>

FIG. 44 is a diagram illustrating a multi-view image encoding devicewhich performs the above-described multi-view image encoding. Asillustrated in FIG. 44 , the multi-view image encoding device 600 has anencoding unit 601, an encoding unit 602, and a multiplexing unit 603.

The encoding unit 601 encodes a base view image to generate a base viewimage encoded stream. The encoding unit 602 encodes a non-base viewimage to generate a non-base view image encoded stream. The multiplexingunit 603 multiplexes the base view image encoded stream generated by theencoding unit 601 and the non-base view image encoded stream generatedby the encoding unit 602 to generate a multi-view image encoded stream.

The base layer image coding unit 101 may be applied as the coding unit601 of the multi-view image coding device 600, and the enhancement layerimage coding unit 102 may be applied as the coding unit 602. In thismanner, it is possible to recognize performance necessary for decodingmore accurately.

<Multi-View Image Decoding Device>

FIG. 45 is a diagram illustrating a multi-view image decoding devicewhich performs the above-described multi-view image decoding. Asillustrated in FIG. 45 , the multi-view image decoding device 610 has andemultiplexing unit 611, a decoding unit 612, and another decoding unit613.

The demultiplexing unit 611 demultiplexes the multi-view image encodedstream obtained by multiplexing the base view image encoded stream andthe non-base view image encoded stream to extract the base view imageencoded stream and the non-base view image encoded stream. The decodingunit 612 decodes the base view image encoded stream extracted by thedemultiplexing unit 611 to obtain the base view image. The decoding unit613 decodes the non-base view image encoded stream extracted by thedemultiplexing unit 611 to obtain the non-base view image.

The base layer image decoding unit 202 may be applied as the decodingunit 612 of the multi-view image decoding device 610, and theenhancement layer image decoding unit 203 may be applied as the decodingunit 613. In this manner, it is possible to recognize performancenecessary for decoding more accurately.

5. Fifth Embodiment

<Computer>

The above-described series of processes can also be performed byhardware and can also be performed by software. When the series ofprocesses is performed by software, a program of the software isinstalled in a computer. Here, the computer includes a computer embeddedin dedicated hardware and, for example, a general personal computercapable of various functions through installation of various programs.

FIG. 46 is a block diagram illustrating an example of a hardwareconfiguration of the computer performing the above-described series ofprocesses according to a program.

In a computer 800 illustrated in FIG. 46 , a central processing unit(CPU) 801, a read-only memory (ROM) 802, and a random access memory(RAM) 803 are connected mutually via a bus 804.

An input and output interface 810 is also connected to the bus 804. Aninput unit 811, an output unit 812, a storage unit 813, a communicationunit 814, and a drive 815 are connected to the input and outputinterface 810.

The input unit 811 is formed by, for example, a keyboard, a mouse, amicrophone, a touch panel, or an input terminal. The output unit 812 isformed by, for example, a display, a speaker, or an output terminal. Thestorage unit 813 is formed by, for example, a hard disk, a RAM disk, ora non-volatile memory. The communication unit 814 is formed by, forexample, a network interface. The drive 815 drives a removable medium821 such as a magnetic disk, an optical disc, a magneto-optical disc, ora semiconductor memory.

In the computer having the above-described configuration, for example,the CPU 801 performs the above-described processes by loading a programstored in the storage unit 813 to the RAM 803 via the input and outputinterface 810 and the bus 804 and executing the program. The RAM 803also appropriately stores data necessary for the CPU 801 to performvarious processes.

For example, a program executed by the computer (the CPU 801) can berecorded in the removable medium 821 such as a package medium to beapplied, in this case, by mounting the removable medium 821 on the drive815, the program can be installed in the storage unit 813 via the inputand output interface 810.

The program can also be supplied via a wired or wireless transmissionmedium such as a local area network, the Internet, or digital satellitebroadcast. In this case, the program can be received by thecommunication unit 814 to be installed in the storage unit 813.

Further, the program can also be installed in advance in the ROM 802 orthe storage unit 813.

Programs executed by the computer may be programs which are processedchronologically in the order described in the present specification ormay be programs which are processed at necessary timings, for example,in parallel or when called.

In the present specification, steps describing a program recorded in arecording medium include not only processes which are performedchronologically in the described order but also processes which areperformed in parallel or individually but not chronologically.

In the present specification, a system means a set of a plurality ofconstituent elements (devices, modules (components), and the like) andall of the constituent elements may be included or may not be includedin the same casing. Accordingly, a plurality of devices accommodated inseparate casings and connected via networks and a single device in whicha plurality of modules are accommodated in a single casing are allsystems.

A configuration described above as a single device (or processing unit)may be divided and configured as a plurality of devices (or processingunits). In contrast, a configuration described above as a plurality ofdevices (or processing units) may be collected and configured as asingle device (or processing unit). Configurations other than theabove-described configurations may, of course, be added to theconfigurations of the devices (or the processing units). Further, aslong as configurations or operations are substantially the same in theentire system, parts of the configurations of certain devices (orprocessing units) may be included in the configurations of the otherdevices (or other processing units).

The preferred embodiments of the present disclosure have been describedabove with reference to the accompanying drawings, whilst the presentdisclosure is not limited to the above examples, of course. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

For example, in the present technology, it is possible to realize acloud computing configuration in which a single function is shared andprocessed jointly by a plurality of devices via networks.

Each step described in the above-described flowcharts can be performedby a single device and can also be shared and performed by a pluralityof devices.

When a plurality of processes are included in a single step, theplurality of processes included in the single step can be performed by asingle device and can also be shared and performed by a plurality ofdevices.

The image encoding device and the image decoding device according to theabove-described embodiments can be applied to various electronic devicessuch as a transmitter or a receiver in delivery of satellite broadcast,a wired broadcast such as a cable TV, or the Internet and delivery to aterminal by cellular communication, a recording device recording animage in a medium such as an optical disc, a magnetic disk, or a flashmemory, or a reproduction device reproducing an image from the storagemedium. Hereinafter, four application examples will be described.

6. Sixth Embodiment

<First Application Example: Television Receiver>

FIG. 47 is a block diagram illustrating an example of a schematicconfiguration of a television device to which the above-describedembodiments are applied. A television device 900 includes an antenna901, a tuner 902, a demultiplexer 903, a decoder 904, a video signalprocessing unit 905, a display unit 906, an audio signal processing unit907, a speaker 908, an external interface (I/F) unit 909, a control unit910, a user interface (I/F) unit 911, and a bus 912.

The tuner 902 extracts a signal of a desired channel from a broadcastsignal received via the antenna 901 and demodulates the extractedsignal. The tuner 902 then outputs an encoded bit stream obtainedthrough the demodulation to the demultiplexer 903. That is, in thetelevision device 900, the tuner 902 serves as a transmission unitconfigured to receive a coding stream in which an image is coded.

The demultiplexer 903 demultiplexes a video stream and an audio streamof a viewing target program from the encoded bit stream and outputs thedemultiplexed streams to the decoder 904. The demultiplexer 903 extractsauxiliary data such as an electronic program guide (EPG) from theencoded bit stream and supplies the extracted data to the control unit910. Also, when the coding bitstream is scrambled, the demultiplexer 903may perform descrambling.

The decoder 904 decodes the video stream and the audio stream input fromthe demultiplexer 903. The decoder 904 outputs video data generatedthrough the decoding process to the video signal processing unit 905.The decoder 904 outputs audio data generated through the decodingprocess to the audio signal processing unit 907.

The video signal processing unit 905 reproduces the video data inputfrom the decoder 904, and causes a video to be displayed on the displayunit 906. In addition, the video signal processing unit 905 may cause anapplication screen supplied via the network to be displayed on thedisplay unit 906. In addition, the video signal processing unit 905 mayperform an additional process, for example, noise elimination, on thevideo data according to settings. Further, the video signal processingunit 905 may generate an image of a graphical user interface (GUI), forexample, a menu, a button or a cursor, and superimpose the generatedimage on an output image.

The display unit 906 is driven by a drive signal supplied from the videosignal processing unit 905 and displays a video or an image on a videoplane of a display device (for example, a liquid crystal display, aplasma display or an organic electro luminescence display (OELD)(organic EL display)).

The audio signal processing unit 907 performs a reproducing process suchas D/A conversion and amplification on audio data input from the decoder904, and causes audio to be output from the speaker 908. In addition,the audio signal processing unit 907 may perform an additional processsuch as noise elimination on the audio data.

The external interface unit 909 is an interface that connects thetelevision device 900 and an external device or the network. Forexample, the video stream or the audio stream received through theexternal interface unit 909 may be decoded by the decoder 904. That is,in the television device 900, the external interface unit 909 serves asa transmission unit configured to receive a coding stream in which animage is coded.

The control unit 910 includes a processor such as a CPU and memoriessuch as a RAM and a ROM. The memories store programs executed by theCPU, program data, EPG data, data acquired via a network, and the like.The programs stored in the memories are read and executed by the CPU,for example, when the television device 900 is activated. The CPUcontrols an operation of the television device 900, for example,according to an operation signal input from the user interface unit 911by executing a program.

The user interface unit 911 is connected to the control unit 910. Theuser interface unit 911 includes, for example, a button and a switchused by the user to operate the television device 900, and a receptionunit of a remote control signal. The user interface unit 911 detects theuser's operation through such a component, generates an operation signaland outputs the generated operation signal to the control unit 910.

The bus 912 connects the tuner 902, the demultiplexer 903, the decoder904, the video signal processing unit 905, the audio signal processingunit 907, the external interface unit 909 and the control unit 910 toone another.

In the television device 900 configured in this manner, the decoder 904has functions of the image decoding device 200 according to theabove-described embodiment. Accordingly, is possible to recognizeperformance necessary for decoding the image more accurately in thetelevision device 900.

<Second Application Example: Mobile Telephone>

FIG. 48 illustrates an exemplary schematic configuration of a mobiletelephone to which the above-described embodiment is applied. A mobiletelephone 920 includes an antenna 921, a communication unit 922, anaudio codec 923, a speaker 924, a microphone 925, a camera unit 926, animage processing unit 927, a demultiplexing unit 928, a recording andreproduction unit 929, a display unit 930, a control unit 931, anoperation unit 932, and a bus 933.

The antenna 921 is connected to the communication unit 922. The speaker924 and the microphone 925 are connected to the audio codec 923. Theoperation unit 932 is connected to the control unit 931. The bus 933connects the communication unit 922, the audio codec 923, the cameraunit 926, the image processing unit 927, the demultiplexing unit 928,the recording and reproduction unit 929, the display unit 930, and thecontrol unit 931 to one another.

The mobile telephone 920 performs operations such as transmission andreception of audio signals, transmission and reception of electronicmail or image data, capturing of images, and recording of data invarious operation modes such as an audio calling mode, a datacommunication mode, a photographing mode, and a video phone mode.

In the audio calling mode, an analog audio signal generated by themicrophone 925 is supplied to the audio codec 923. The audio codec 923converts the analog audio signal into the audio data and performs A/Dconversion and compression of the converted audio data. Therefore, theaudio codec 923 outputs the compressed audio data to the communicationunit 922. The communication unit 922 codes and modulates the audio dataand generates a transmission signal. Therefore, the communication unit922 transmits the generated transmission signal to a base station (notillustrated) through the antenna 921. In addition, the communicationunit 922 amplifies a wireless signal received through the antenna 921,performs frequency conversion thereon, and acquires a received signal.Therefore, the communication unit 922 demodulates and decodes thereceived signal to generate audio data, and outputs the generated audiodata to the audio codec 923. The audio codec 923 extends the audio dataand performs D/A conversion thereon, and generates an analog audiosignal. Therefore, the audio codec 923 supplies the generated audiosignal to the speaker 924 and causes audio to be output.

In addition, in the data communication mode, for example, the controlunit 931 generates text data of E-mail according to the user's operationthrough the operation unit 932. In addition, the control unit 931 causestext to be displayed on the display unit 930. In addition, the controlunit 931 generates E-mail data according to a transmission instructionby the user through the operation unit 932, and outputs the generated.E-mail data to the communication unit 922. The communication unit 922codes and modulates the E-mail data and generates a transmission signal.Therefore, the communication unit 922 transmits the generatedtransmission signal to the base station (not illustrated) through theantenna 921. In addition, the communication unit 922 amplifies awireless signal received through the antenna 921, performs frequencyconversion thereon, and acquires a received signal. Therefore, thecommunication unit 922 demodulates and decodes the received signal,restores the E-mail data, and outputs the restored E-mail data to thecontrol unit 931. The control unit 931 causes content of E-mail to bedisplayed on the display unit 930 and supplies the E-mail data to therecording and reproduction unit 929 and causes the data to be written inthe storage medium.

The recording and reproduction unit 929 includes an arbitrary readableand writable storage medium. For example, the storage medium may be abuilt-in storage medium such as a RAM or a flash memory, or a storagemedium of an external mounting type such as a hard disk, a magneticdisk, a magneto optical disk, an optical disc, a Universal Serial Bus(USB) memory or a memory card.

In addition, in the photographing mode, for example, the camera unit 926captures an image of a subject, generates image data, and outputs thegenerated image data to the image processing unit 927. The imageprocessing unit 927 codes the image data input from the camera unit 926,supplies the coding stream to the recording and reproduction unit 929,and causes the stream to be written in the storage medium.

Further, in an image display mode, the recording and reproduction unit929 reads the coding stream recorded in the storage medium and outputsthe read stream to the image processing unit 927. The image processingunit 927 decodes the coding stream input from the recording andreproduction unit 929, supplies the image data to the display unit 930,and causes the image to be displayed.

In addition, in the television phone mode, for example, thedemultiplexing unit 928 multiplexes the video stream coded by the imageprocessing unit 927 and the audio stream input from the audio codec 923and outputs the multiplexed stream to the communication unit 922. Thecommunication unit 922 codes and modulates the stream and generates atransmission signal. Therefore, the communication unit 922 transmits thegenerated transmission signal to the base station (not illustrated)through the antenna 921. In addition, the communication unit 922amplifies a wireless signal received through the antenna 921, performsfrequency conversion thereon, and acquires a received signal. The codingbitstream may be included in the transmission signal and the receivedsignal. Therefore, the communication unit 922 demodulates and decodesthe received signal, restores the stream, and outputs the restoredstream to the demultiplexing unit 928. The demultiplexing unit 928separates the video stream and the audio stream from the input stream,and outputs the video stream to the image processing unit 927 and theaudio stream to the audio codec 923. The image processing unit 927decodes the video stream and generates the video data. The video data issupplied to the display unit 930, and a series of images is displayed onthe display unit 930. The audio codec 923 extends the audio stream andperforms D/A conversion thereon, and generates an analog audio signal.Therefore, the audio codec 923 supplies the generated audio signal tothe speaker 924 and causes audio to be output.

In the mobile telephone 920 configured in this manner, the imageprocessing unit 927 has functions of the image coding device 100 or theimage decoding device 200 according to the above-described embodiment.Accordingly, it is possible to recognize performance necessary fordecoding more accurately in the mobile telephone 920.

<Third Application Example: Recording and Reproduction Device>

FIG. 49 illustrates an exemplary schematic configuration of a recordingand reproduction device to which the above-described embodiment isapplied. A recording and reproduction device 940 codes, for example,received audio data and video data of a broadcast program, and recordsthe result in the recording medium. In addition, the recording andreproduction device 940 may code the audio data and video data acquiredfrom, for example, another device and record the result in the recordingmedium. In addition, the recording and reproduction device 940reproduces data recorded in the recording medium through a monitor and aspeaker according to, for example, the user's instruction. In this case,the recording and reproduction device 940 decodes the audio data and thevideo data.

The recording and reproduction device 940 includes a tuner 941, anexternal interface (I/F) unit 942, an encoder 943, a hard disk drive(HDD) 944, a disk drive 945, a selector 946, a decoder 947, an on-screendisplay (OSD) 948, a control unit 949, and a user interface (I/F) unit950.

The tuner 941 extracts a desired channel signal from a broadcast signalreceived through the antenna (not illustrated), and demodulates theextracted signal. Therefore, the tuner 941 outputs the coding bitstreamobtained by demodulation to the selector 946. That is, the tuner 941serves as a transmission unit in the recording and reproduction device940.

The external interface unit 942 is an interface that connects therecording and reproduction device 940 and the external device or thenetwork. The external interface unit 942 may be, for example, anInstitute of Electrical and Electronic Engineers (IEEE) 1394 interface,a network interface, a USB interface, or a flash memory interface. Forexample, the video data and audio data received through the externalinterface unit 942 are input to the encoder 943. That is, the externalinterface unit 942 serves as a transmission unit in the recording andreproduction device 940.

When the video data and audio data input from the external interfaceunit 942 are not coded, the encoder 943 codes the video data and theaudio data. Therefore, the encoder 943 outputs the coding bitstream tothe selector 946.

The HDD 944 records the coding bitstream in which content data such as avideo and audio is compressed, various programs and other data in aninternal hard disk. In addition, when the video and the audio arereproduced, the HDD 944 reads this data from the hard disk.

The disk drive 945 records and reads data in the installed recordingmedium. The recording medium installed in the disk drive 945 may be, forexample, a digital versatile disc (DVD) disc (such as a DVD-Video, aDVD-random access memory (DVD-RAM), a DVD-Recordable (DVD-R), aDVD-Rewritable (DVD-RW), DVD+Recordable (DVD+R), and a DVD+Rewritable(DVD+RW)) or a Blu-ray (registered trademark) disc.

When a video and audio are recorded, the selector 946 selects the codingbitstream input from the tuner 941 or the encoder 943, and outputs theselected coding bitstream to the HDD 944 or the disk drive 945. Inaddition, when a video and audio are reproduced, the selector 946outputs the coding bitstream input from the HDD 944 or the disk drive945 to the decoder 947.

The decoder 947 decodes the coding bitstream and generates video dataand audio data. Therefore, the decoder 947 outputs the generated videodata to the OSD 948. In addition, the decoder 947 outputs the generatedaudio data to an external speaker.

The OSD 948 reproduces the video data input from the decoder 947 anddisplays a video. In addition, the OSD 948 may superimpose an image of aGUI, for example, a menu, a button or a cursor, on the video to bedisplayed.

The control unit 949 includes a processor such as a CPU and memoriessuch as a RAM and a ROM. The memories store programs executed by theCPU, program data, and the like. The programs stored in the memories areread and executed by the CPU, for example, when the recording andreproduction device 940 is activated. The CPU controls an operation ofthe recording and reproduction device 940, for example, according to anoperation signal input from the user interface unit 950 by executing aprogram.

The user interface unit 950 is connected to the control unit 949. Theuser interface unit 950 includes, for example, a button and a switchused by the user to operate the recording and reproduction device 940,and a reception unit of a remote control signal. The user interface unit950 detects the user's operation through such a component, generates anoperation signal and outputs the generated operation signal to thecontrol unit 949.

In the recording and reproduction device 940 configured in this manner,the encoder 943 has functions of the image coding device 100 accordingto the above-described embodiment. In addition, the decoder 947 hasfunctions of the image decoding device 200 according to theabove-described embodiment. Accordingly, it is possible to recognizeperformance necessary for decoding the image more accurately in therecording and reproduction device 940.

<Fourth Application Example: Imaging Device>

FIG. 50 illustrates an exemplary schematic configuration of an imagingdevice to which the above-described embodiment is applied. An imagingdevice 960 generates an image captured of a subject, codes image data,and records the data in the recording medium.

The imaging device 960 includes an optical block 961, an imaging unit962, a signal processing unit 963, an image processing unit 964, adisplay unit 965, an external interface (I/F) unit 966, a memory unit967, a media drive 968, an OSD 969, a control unit 970, a user interface(I/F) unit 971, and a bus 972.

The optical block 961 is connected to the imaging unit 962. The imagingunit 962 is connected to the signal processing unit 963. The displayunit 965 is connected to the image processing unit 964. The userinterface unit 971 is connected to the control unit 970. The bus 972connects the image processing unit 964, the external interface unit 966,the memory unit 967, the media drive 968, the OSD 969, and the controlunit 970 to one another.

The optical block 961 includes a focus lens and an aperture mechanism.The optical block 961 causes an optical image of the subject to beformed on an imaging plane of the imaging unit 962. The imaging unit 962includes an image sensor such as a charge coupled device (CCD) or acomplementary metal oxide semiconductor (CMOS), and converts the opticalimage formed on the imaging plane into an image signal as an electricalsignal by photoelectric conversion. Therefore, the imaging unit 962outputs the image signal to the signal processing unit 963.

The signal processing unit 963 performs various camera signal processessuch as knee correction, gamma correction, and color correction on theimage signal input from the imaging unit 962. The signal processing unit963 outputs image data after the camera signal process to the imageprocessing unit 964.

The image processing unit 964 codes the image data input from the signalprocessing unit 963 and generates coded data. Therefore, the imageprocessing unit 964 outputs the generated coded data to the externalinterface unit 966 or the media drive 968. In addition, the imageprocessing unit 964 decodes the coded data input from the externalinterface unit 966 or the media drive 968 and generates image data.Therefore, the image processing unit 964 outputs the generated imagedata to the display unit 965. In addition, the image processing unit 964may output the image data input from the signal processing unit 963 andcause the image to be displayed on the display unit 965. In addition,the image processing unit 964 may superimpose the image to be output tothe display unit 965 on display data acquired from the OSD 969.

The OSD 969 generates an image of a GUI, for example, a menu, a buttonor a cursor, and outputs the generated image to the image processingunit 964.

The external interface unit 966 is configured as, for example, a USBinput and output terminal. The external interface unit 966 connects theimaging device 960 and a printer, for example, when the image isprinted. In addition, a drive is connected to the external interfaceunit 966 as necessary. A removable medium, for example, a magnetic diskor an optical disc, is installed in the drive, and a program read fromthe removable medium may be installed in the imaging device 960.Further, the external interface unit 966 may be configured as a networkinterface that is connected to the network such as the LAN or theInternet. That is, the external interface unit 966 serves as atransmission unit in the imaging device 960.

The recording medium installed in the media drive 968 may be anyreadable and writable removable medium, for example, a magnetic disk, amagneto optical disk, an optical disc, or a semiconductor memory. Inaddition, the recording medium may be fixedly installed in the mediadrive 968, and a non-portable storage unit, for example, a built-in harddisk drive or a solid state drive (SSD), may be configured.

The control unit 970 includes a processor such as a CPU and memoriessuch as a RAM and a ROM. The memories store programs executed by theCPU, program data, and the like. The programs stored in the memories areread and executed by the CPU, for example, when the imaging device 960is activated. The CPU controls an operation of the imaging device 960,for example, according to an operation signal input from the userinterface unit 971 by executing a program.

The user interface unit 971 is connected to the control unit 970. Theuser interface unit 971 includes, for example, a button and a switchused by the user to operate the imaging device 960, and the like. Theuser interface unit 971 detects the user's operation through such acomponent, generates an operation signal and outputs the generatedoperation signal to the control unit 970.

In the imaging device 960 configured in this manner, the imageprocessing unit 964 has functions of the image coding device 100 and theimage decoding device 200 according to the above-described embodiment.Accordingly, it is possible to recognize performance necessary fordecoding more accurately in the imaging device 960.

7. Seventh Embodiment

<Application Example of Scalable Coding: First System>

Next, a detailed use example of scalably coded data that is scalablycoded (layered (image) coding) will be described. The scalable coding isused to select data to be transmitted, for example, as exemplified inFIG. 51 .

In a data transmission system 1000 illustrated in FIG. 51 , a deliveryserver 1002 reads scalably coded data stored in a scalably coded datastorage unit 1001 and delivers the data to a terminal device such as apersonal computer 1004, an AV instrument 1005, a tablet device 1006, ora mobile telephone 1007 via a network 1003.

In this case, the distribution server 1002 selects and transmits codeddata of appropriate quality according to a capability of the terminaldevice, a communication environment or the like. Even when thedistribution server 1002 transmits unnecessarily high quality data, notonly may an image of high image quality not be obtained in the terminaldevice, but a delay or an overflow may be caused. In addition, there isconcern of a communication band being unnecessarily occupied, and a loadon the terminal device unnecessarily increasing. On the other hand, evenwhen the distribution server 1002 transmits unnecessarily low qualitydata, there is concern of an image of sufficient image quality not beingobtained in the terminal device. Therefore, the distribution server 1002appropriately reads the scalably coded data stored in the scalably codeddata storage unit 1001 as coded data of appropriate quality andtransmits the data according to a capability of the terminal device, acommunication environment or the like.

For example, the scalably coded data storage unit 1001 stores scalablycoded data (BL+EL) 1011 that is scalably coded. The scalably coded data(BL+EL) 1011 is coded data including both the base layer and theenhancement layer, and is data from which both the image of the baselayer and the image of the enhancement layer can be obtained bydecoding.

The delivery server 1002 selects an appropriate layer according to acapability of the terminal device configured to transmit data, acommunication environment or the like, and reads data of the layer. Forexample, the delivery server 1002 reads the scalably coded data (BL+EL)1011 of high quality from the scalably coded data storage unit 1001 forthe personal computer 1004 or the tablet device 1006 having a highprocessing capacity and transmits the data without change. On the otherhand, for example, the delivery server 1002 extracts data of the baselayer from the scalably coded data (BL+EL) 1011 for the AV instrument1005 or the mobile telephone 1007 having a low processing capacity, andtransmits the data as scalably coded data (BL) 1012 that has the samecontent as the scalably coded data (BL+EL) 1011 but has lower qualitythan the scalably coded data (BL+EL) 1011.

When such scalably coded data is used, since it is possible to easilyadjust an amount of data, it is possible to suppress a delay or anoverflow from occurring, and suppress an unnecessary load on theterminal device or a communication medium from increasing. In addition,since redundancy between layers is reduced, it is possible to decreasethe amount of data in the scalably coded data (BL+EL) 1011, compared towhen coded data of each layer is set as separate data. Therefore, it ispossible to use a storage area of the scalably coded data storage unit1001 with higher efficiency.

Also, similarly to the personal computer 1004 to the mobile telephone1007, since various devices can be applied to the terminal device,hardware performance of the terminal device is different according tothe device. In addition, since there are various applications that areexecuted by the terminal device, a capability of software thereof isdifferent. Further, as the network 1003 serving as a communicationmedium, any communication channel network including either or both ofwired and wireless communication, for example, the Internet or a localarea network (LAN) may be applied, and a data transmission capabilitythereof is different. Further, the capability may be changed accordingto other communication or the like.

Therefore, before data transmission starts, the delivery server 1002 mayperform communication with the terminal device serving as a datatransmission destination, and obtain information on a capability of theterminal device such as hardware performance of the terminal device orperformance of the application (software) executed by the terminaldevice, and information on a communication environment such as anavailable bandwidth of the network 1003. Therefore, the delivery server1002 may select an appropriate layer based on the information obtainedherein.

Also, extraction of the layer may be performed in the terminal device.For example, the personal computer 1004 may decode the transmittedscalably coded data (BL+EL) 1011, display the image of the base layer,or display the image of the enhancement layer. In addition, for example,after extracting the scalably coded data (BL) 1012 of the base layerfrom the transmitted scalably coded data (BL+EL) 1011, the personalcomputer 1004 may store it, transfer it to another device, decode it anddisplay the image of the base layer.

It is needless to say that the scalably coded data storage unit 1001,the delivery server 1002, the network 1003, and the number of terminaldevices are arbitrary. In addition, while the example in which thedelivery server 1002 transmits data to the terminal device has beendescribed above, the use example is not limited thereto. The datatransmission system 1000 may be applied to any system as long as thesystem selects and transmits an appropriate layer according to acapability of the terminal device, a communication environment or thelike when coded data that is scalably coded is transmitted to theterminal device.

Therefore, similarly to an application to the layered coding and layereddecoding described above with reference to FIGS. 1 to 42 , when thepresent technology is applied to the data transmission system 1000 inFIG. 51 , it is possible to obtain the same effects that were describedabove with reference to FIGS. 1 to 42 .

<Application Example of Scalable Coding: Second System>

In addition, the scalable coding is used for transmission through aplurality of communication media, for example, as exemplified in FIG. 52.

In a data transmission system 1100 illustrated in FIG. 52 , a broadcaststation 1101 transmits scalably coded data of the base layer (BL) 1121through terrestrial broadcast 1111. In addition, the broadcast station1101 transmits scalably coded data of the enhancement layer (EL) 1122via any network 1112 formed of either or both of wired and wirelesscommunication networks (for example, data is packetized andtransmitted).

A terminal device 1102 includes a function of receiving the terrestrialbroadcast 1111 that is broadcast from the broadcast station 1101, andreceives the scalably coded data of the base layer (BL) 1121 transmittedthrough the terrestrial broadcast 1111. In addition, the terminal device1102 further includes a communication function of performingcommunication via the network 1112, and receives the scalably coded dataof the enhancement layer (EL) 1122 transmitted via the network 1112.

The terminal device 1102 decodes the scalably coded data of the baselayer (BL) 1121 acquired through the terrestrial broadcast 1111according to, for example, the user's instruction, obtains and storesthe image of the base layer, and transmits the result to other device.

In addition, the terminal device 1102 synthesizes the scalably codeddata of the base layer (BL) 1121 acquired through the terrestrialbroadcast 1111 and the scalably coded data of the enhancement layer (EL)1122 acquired via the network 1112 according to, for example, the user'sinstruction, obtains scalably coded data (B+EL), decodes the data,obtains and stores the image of the enhancement layer, and transmits theresult to other device.

As described above, the scalably coded data may be transmitted through,for example, a communication medium different for each layer. Therefore,it is possible to deliver a load and suppress a delay or an overflowfrom occurring.

In addition, according to circumstances, a communication medium used fortransmission may be selected for each layer. For example, the scalablycoded data of the base layer (BL) 1121 having a relatively large amountof data may be transmitted through a communication medium having a widebandwidth, and the scalably coded data of the enhancement layer (EL)1122 having a relatively small amount of data may be transmitted througha communication medium having a narrow bandwidth. In addition, forexample, the communication medium for transmitting the scalably codeddata of the enhancement layer (EL) 1122 may be switched to the network1112 or the terrestrial broadcast 1111 according to an availablebandwidth of the network 1112. It is needless to say that this issimilar for data of any layer.

When control is performed in this manner, it is possible to furthersuppress a load on data transmission from increasing.

It is needless to say that the number of layers is arbitrary, and thenumber of communication media used for transmission is also arbitrary.In addition, the number of terminal devices 1102 serving as a datadelivery destination is also arbitrary. Further, while the example ofbroadcast from the broadcast station 1101 has been described above, theuse example is not limited thereto. The data transmission system 1100may be applied to any system as long as the system divides the codeddata that is scalably coded into a plurality of pieces of data using alayer as a unit, and transmits the data through a plurality of lines.

Therefore, similarly to an application to the layered coding and layereddecoding described above with reference to FIGS. 1 to 42 , when thepresent technology is applied to the data transmission system 1100 in 43described above, it is possible to obtain the same effects that weredescribed above with reference to FIGS. 1 to 42 .

<Application Example of Scalable Coding: Third System>

Also, the scalable coding is used for storing coded data, for example,as exemplified in FIG. 53 .

In an imaging system 1200 illustrated in FIG. 53 , an imaging device1201 scalably codes image data obtained by capturing an image of asubject 1211, and supplies the result to a scalably coded data storagedevice 1202 as scalable coded data (BL+EL) 1221.

The scalably coded data storage device 1202 stores the scalably codeddata (BL+EL) 1221 supplied from the imaging device 1201 with qualityaccording to circumstances. For example, at normal times, the scalablycoded data storage device 1202 extracts data of the base layer from thescalably coded data (BL+EL) 1221, and stores it as scalably coded dataof the base layer (BL) 1222 having a small amount of data with lowquality. On the other hand, for example, in times of significance, thescalably coded data storage device 1202 directly stores the scalablycoded data (BL+EL) 1221 having a large amount of data with high quality.

In this manner, since the scalably coded data storage device 1202 maysave the image with high image quality only as necessary, it is possibleto suppress an amount of data from increasing and it is possible toincrease utilization efficiency of a storage area while suppressing avalue of an image from decreasing due to image quality degradation.

For example, the imaging device 1201 is a surveillance camera. Whenthere is no monitoring target (for example, an intruder) in a capturedimage (at normal times), since content of the captured image is highlylikely to be unimportant, reducing an amount of data has a priority andthe image data (scalably coded data) is stored in low quality. On theother hand, when a monitoring target is shown in the captured image asthe subject 1211 (in times of significance), since content of thecaptured image is highly likely to be important, image quality has apriority, and the image data (scalably coded data) is stored in highquality.

Also, the scalably coded data storage device 1202 may determine normaltimes and times of significance by, for example, analyzing the image.Alternatively, the imaging device 1201 may perform determination andtransmit the determination result to the scalably coded data storagedevice 1202.

Also, a reference for determining normal times and times of significanceis arbitrary, and content of the image serving as the determinationreference is arbitrary. It is needless to say that a condition otherthan content of the image may be set as the determination reference. Forexample, the determination reference may be switched according to amagnitude, a waveform or the like of recorded audio, switched atpredetermined time intervals or switched according to an instructionfrom the outside such as the user's instruction.

In addition, while the example in which two states, at normal times andin times of significance, are switched has been described above, thenumber of states is arbitrary. For example, three or more states such asnormal times, times of slight significance, times of significance, andtimes of great significance may be switched. However, a maximum numberof states to be switched depends on the number of layers of the scalablycoded data.

In addition, the imaging device 1201 may determine the number of layersof scalable coding according to the state. For example, at normal times,the imaging device 1201 may generate the scalably coded data of the baselayer (BL) 1222 having a small amount of data with low quality andsupply the generated data to the scalably coded data storage device1202. In addition, for example, in times of significance, the imagingdevice 1201 may generate the scalably coded data (BL+EL) 1221 of thebase layer having a large amount of data with high quality and supplythe generated data to the scalably coded data storage device 1202.

While the surveillance camera has been exemplified above, an applicationof the imaging system 1200 is arbitrary, and is not limited to thesurveillance camera.

Therefore, similarly to an application to the layered coding and layereddecoding described above with reference to FIGS. 1 to 42 , when thepresent technology is applied to the imaging system 1200 in FIG. 53 , itis possible to obtain the same effects that were described above withreference to FIGS. 1 to 42 .

Also, the present technology may be applied to, for example, HTTPstreaming such as MPEG DASH through which appropriate data is selectedfrom among a plurality of previously prepared pieces of coded datahaving different resolutions in units of segments and is used. That is,information on coding or decoding may be shared among the plurality ofpieces of coded data.

8. Eighth Embodiment

<Other Examples>

Although the examples of devices, systems, and the like to which thepresent technology is applied have been described above, the presenttechnology is not limited thereto, and can be implemented as anyconfiguration mounted in the devices or devices constituting thesystems, for example, processors in the form of system large scaleintegration (LSI), modules that use a plurality of processors, unitsthat use a plurality of modules, sets obtained by further adding otherfunctions to the units (i.e., a partial configuration of the devices),and the like.

<Video Set>

An example in which the present technology is implemented as a set willbe described with reference to FIG. 54 . FIG. 54 illustrates an exampleof a schematic configuration of a video set to which the presentdisclosure is applied.

As electronic apparatuses have gradually become multifunctional inrecent years, when some configurations of each apparatus are preparedfor sale, provision, and the like in the stage of development andmanufacturing, there are not only cases in which such an apparatus isconfigured to have one function, but also many cases in which aplurality of configurations having relevant functions are combined andimplemented as one set with the plurality of functions.

The video set 1300 illustrated in FIG. 54 is configured to bemultifunctional as described above by combining devices having functionsof encoding and decoding (which may have either or both of thefunctions) of images with devices having other functions relating to theforegoing functions.

As illustrated in FIG. 54 , the video set 1300 has a module groupincluding a video module 1311, an external memory 1312, a powermanagement module 1313, a frontend module 1314 and the like, and deviceshaving relevant functions such as connectivity 1321, a camera 1322, asensor 1323, and the like.

A module is a form of a component in which several related componentialfunctions are gathered to provide a cohesive function. A specificphysical configuration is arbitrary; however, it is considered to be anintegration in which, for example, a plurality of processors each havingfunctions, electronic circuit elements such as a resistor and acapacitor, and other devices are disposed on a circuit board. Inaddition, making a new module by combining a module with another module,a processor, or the like is also considered.

In the example of FIG. 54 , the video module 1311 is a combination ofconfigurations with functions relating to image processing, and has anapplication processor, a video processor, a broadband modem 1333, and anRF module 1334.

A processor is a semiconductor chip integrated with a configurationhaving predetermined functions using System-On-Chip (SoC), and is alsoreferred to as, for example, system large scale integration (LSI), orthe like. The configuration having a predetermined function may be alogic circuit (hardware configuration), may be, along with CPU, a ROM,and a RAM, a program that is executed by using the elements (softwareconfiguration), or may be a combination of both configurations. Forexample, a processor may have a logic circuit, a CPU, a ROM, a RAM, andthe like and may realize some functions with the logic circuit (hardwareconfiguration), or may realize the other functions with a programexecuted by the CPU (software configuration).

The application processor 1331 of FIG. 54 is a processor that executesan application relating to image processing. The application executed bythe application processor 1331 can not only perform an arithmeticprocess but can also control a configuration internal and external tothe video module 1311, for example, the video processor 1332 whennecessary in order to realize predetermined functions.

The video processor 1332 is a processor having a function relating to(one or both of) encoding and decoding of images.

The broadband modem 1333 digitally modulates data (a digital signal)transmitted through wired or wireless (or both) broadband communicationthat is performed through a broadband line such as the Internet or apublic telephone network, converts the result into an analog signal, anddemodulates the analog signal received through the broadbandcommunication and converts the result into data (digital signal). Thebroadband modem 1333 processes any information, for example, image dataprocessed by the video processor 1332, a stream in which image data iscoded, an application program, or setting data.

The RF module 1334 is a module which performs frequency conversion,modulation and demodulation, amplification, a filtering process, and thelike on a radio frequency (RF) signal transmitted and received via anantenna. For example, the RF module 1334 generates an RF signal byperforming frequency conversion and the like on a baseband signalgenerated by the broadband modem 1333. In addition, the RF module 1334,for example, generates a baseband signal by performing frequencyconversion and the like on an RF signal received via the frontend module1314.

Note that, as indicated by the dashed line 1341 in FIG. 54 , theapplication processor 1331 and the video processor 1332 may beintegrated to constitute one processor.

The external memory 1312 is a module that is provided outside the videomodule 1311, having a storage device used by the video module 1311. Thestorage device of the external memory 1312 may be realized with anyphysical configuration, but is generally used when large amounts of datasuch as image data in units of frames are stored, and thus it isdesirable to realize the storage device with a relatively inexpensiveand high-capacity semiconductor memory, for example, a dynamic randomaccess memory (DRAM).

The power management module 1313 manages and controls power supply tothe video module 1311 (each constituent element inside the video module1311).

The frontend module 1314 is a module which provides the RF module 1334with a frontend function (serving as a circuit of a transmitting andreceiving end on an antenna side). The frontend module 1314 has, forexample, an antenna unit 1351, a filter 1352, and an amplifying unit1353 as illustrated in FIG. 54 .

The antenna unit 1351 is configured with an antenna which transmits andreceives wireless signals and peripherals thereof. The antenna unit 1351transmits a signal supplied from the amplifying unit 1353 as a radiosignal and supplies a received radio signal to the filter 1352 as anelectric signal (RF signal). The filter 1352 performs a filteringprocess or the like on the RF signal received via the antenna unit 1351and supplies the processed RF signal to the RF module 1334. Theamplifying unit 1353 amplifies an RF signal supplied from the RF module1334, and supplies the signal to the antenna unit 1351.

The connectivity 1321 is a module having a function relating toconnection to the outside. A physical configuration of the connectivity1321 is arbitrary. The connectivity 1321 has, for example, aconfiguration with a communication function other than that of acommunication standard to which the broadband modem 1333 corresponds, anexternal input and output terminal, or the like.

For example, the connectivity 1321 may have a communicating functionthat is based on a wireless communication standard such as Bluetooth (aregistered trademark), IEEE 802.11 (for example, Wireless Fidelity(Wi-Fi; a registered trademark), near field communication (NFC), orInfrared Data Association (IrDA), an antenna which transmits andreceives signals based on the standard, or the like. In addition, theconnectivity 1321 may have, for example, a module having a communicatingfunction based on a wired communication standard such as UniversalSerial Bus (USB), or High-Definition Multimedia Interface (HDMI; aregistered trademark), or a terminal based on the standard. Furthermore,the connectivity 1321 may have, for example, another data (signal)transmitting function of an analog input and output terminal or thelike.

Note that the connectivity 1321 may be set to include a device servingas a data (signal) transmission destination. For example, theconnectivity 1321 may be set to have a drive (including a drive not onlyof a removable medium but also of a hard disk, a solid-state drive(SSD), a network-attached storage (NAS), or the like) which reads andwrites data with respect to a recording medium such as a magnetic disk,an optical disc, a magneto-optical disc, or a semiconductor memory. Inaddition, the connectivity 1321 may be set to have an image or audiooutput device (a monitor, a speaker, or the like).

The camera 1322 is a module having a function of capturing a subject andobtaining image data of the subject. Image data obtained from capturingby the camera 1322 is, for example, supplied to and encoded by the videoprocessor 1332.

The sensor 1323 is a module having arbitrary sensing functions of, forexample, a sound sensor, an ultrasound sensor, a light sensor, anilluminance sensor, an infrared sensor, an image sensor, a rotationsensor, an angle sensor, an angular velocity sensor, a speed sensor, anacceleration sensor, an inclination sensor, a magnetic identificationsensor, a shock sensor, a temperature sensor, and the like. Datadetected by the sensor 1323 is, for example, supplied to the applicationprocessor 1331 and used by an application or the like.

The configurations described as modules above may be realized asprocessors, or conversely the configurations described as processors maybe realized as modules.

In the video set 1300 with the configuration described above, thepresent technology can be applied to the video processor 1332 as will bedescribed below. Thus, the video set 1300 can be implemented as a set towhich the present technology is applied.

<Example of a Configuration of a Video Processor>

FIG. 55 illustrates an example of a schematic configuration of the videoprocessor 1332 (of FIG. 54 ) to which the present technology is applied.

In the example of FIG. 55 , the video processor 1332 has a function ofreceiving a video signal and audio signal input and coding the inputusing a predetermined scheme, and a function of decoding the coded videodata and audio data and reproducing and outputting the video signal andaudio signal.

As illustrated in FIG. 55 , the video processor 1332 has a video inputprocessing unit 1401, a first image enlarging and reducing unit 1402, asecond image enlarging and reducing unit 1403, a video output processingunit 1404, a frame memory 1405, and a memory control unit 1406. Inaddition, the video processor 1332 has an encoding/decoding engine 1407,video elementary stream (ES) buffers 1408A and 1408B, and audio ESbuffers 1409A and 1409B. Furthermore, the video processor 1332 has anaudio encoder 1410, an audio decoder 1411, a multiplexer (MUX) 1412, ademultiplexer (DMUX) 1413, and a stream buffer 1414.

The video input processing unit 1401 acquires a video signal input from,for example, the connectivity 1321 (FIG. 54 ), and converts the signalinto digital image data. The first image enlarging and reducing unit1402 performs format conversion, an image enlarging or reducing processor the like on image data. The second image enlarging and reducing unit1403 performs an image enlarging or reducing process on the image dataaccording to the format of a destination to which the data is output viathe video output processing unit 1404, or performs format conversion, animage enlarging or reducing process or the like in the same manner asthe first image enlarging and reducing unit 1402. The video outputprocessing unit 1404 performs format conversion, conversion into ananalog signal, or the like on image data, and outputs the data to, forexample, the connectivity 1321 as a reproduced video signal.

The frame memory 1405 is a memory for image data shared by the videoinput processing unit 1401, the first image enlarging and reducing unit1402, the second image enlarging and reducing unit 1403, the videooutput processing unit 1404, and the encoding/decoding engine 1407. Theframe memory 1405 is realized as a semiconductor memory, for example, aDRAM, or the like.

The memory control unit 1406 receives a synchronization signal from theencoding/decoding engine 1407 and controls access to the frame memory1405 for writing and reading according to an access schedule to theframe memory 1405 which is written in an access management table 1406A.The access management table 1406A is updated by the memory control unit1406 according to processes executed in the encoding/decoding engine1407, the first image enlarging and reducing unit 1402, the second imageenlarging and reducing unit 1403, and the like.

The encoding/decoding engine 1407 performs an encoding process of imagedata and a decoding process of a video stream that is data obtained byencoding image data. For example, the encoding/decoding engine 1407encodes image data read from the frame memory 1405, and sequentiallywrites the data in the video ES buffer 1408A as video streams. Inaddition, for example, the encoding/decoding engine 1407 sequentiallyreads video streams from the video ES buffer 1408B, and sequentiallywrites the data in the frame memory 1405 as image data. Theencoding/decoding engine 1407 uses the frame memory 1405 as a work areafor such encoding and decoding. In addition, the encoding/decodingengine 1407 outputs a synchronization signal to the memory control unit1406 at a timing at which, for example, a process on each micro block isstarted.

The video ES buffer 1408A buffers a video stream generated by theencoding/decoding engine 1407 and supplies the stream to the multiplexer(MUX) 1412. The video ES buffer 1408B buffers a video stream suppliedfrom the demultiplexer (DMUX) 1413 and supplies the stream to theencoding/decoding engine 1107.

The audio ES buffer 1409A buffers an audio stream generated by an audioencoder 1410 and supplies the stream to the multiplexer (MUX) 1412. Theaudio ES buffer 1409B buffers an audio stream supplied from thedemultiplexer (DMUX) 1413 and supplies the stream to an audio decoder1411.

The audio encoder 1410, for example, digitally converts an audio signalinput from, for example, the connectivity 1321 or the like, and encodesthe signal in a predetermined scheme, for example, an MPEG audio scheme,an AudioCode number 3 (AC3) scheme, or the like. The audio encoder 1410sequentially writes audio streams that are data obtained by encodingaudio signals in the audio ES buffer 1409A. The audio decoder 1411decodes an audio stream supplied from the audio ES buffer 1409B,performs conversion into an analog signal, for example, and supplies thesignal to, for example, the connectivity 1321 or the like as areproduced audio signal.

The multiplexer (MUX) 1412 multiplexes a video stream and an audiostream. A method for this multiplexing (i.e., a format of a bit streamgenerated from multiplexing) is arbitrary. In addition, duringmultiplexing, the multiplexer (MUX) 1412 can also add predeterminedheader information or the like to a bit stream. That is to say, themultiplexer (MUX) 1412 can convert the format of a stream throughmultiplexing. By multiplexing a video stream and an audio stream, forexample, the multiplexer (MUX) 1412 converts the streams into atransport stream that is a bit stream of a format for transport. Inaddition, by multiplexing a video stream and an audio stream, forexample, the multiplexer (MUX) 1412 converts the streams into data of afile format for recording (file data).

The demultiplexer (DMUX) 1413 demultiplexes a bit stream obtained bymultiplexing a video stream and an audio stream using a method whichcorresponds to the multiplexing performed by the multiplexer (MUX) 1412.That is to say, the demultiplexer (DMUX) 1413 extracts a video streamand an audio stream from a bit stream read from the stream buffer 1414(separates the bit stream into the video stream and the audio stream).The demultiplexer (DMUX) 1413 can convert the format of a stream throughdemultiplexing (inverse conversion to conversion by the multiplexer(MUX) 1412). For example, the demultiplexer (DMUX) 1413 can acquire atransport stream supplied from, for example, the connectivity 1321, thebroadband modem 1333, or the like via the stream buffer 1414, andconvert the stream into a video stream and an audio stream throughdemultiplexing. In addition, for example, the demultiplexer (DMUX) 1413can acquire file data read from various recording media by, for example,the connectivity 1321 via the stream buffer 1414, and convert the datainto a video stream and an audio stream through demultiplexing.

The stream buffer 1414 buffers bit streams. For example, the streambuffer 1414 buffers a transport stream supplied from the multiplexer(MUX) 1412, and supplies the stream to, for example, the connectivity1321, the broadband modem 1333, or the like at a predetermined timing orbased on a request from outside or the like.

In addition, for example, the stream buffer 1414 buffers file datasupplied from the multiplexer (MUX) 1412, and supplies the data to, forexample, the connectivity 1321 or the like at a predetermined timing orbased on a request from outside or the like to cause the data to berecorded on any of various kinds of recording media.

Furthermore, the stream buffer 1414 buffers a transport stream acquiredvia, for example, the connectivity 1321, the broadband modem 1333, orthe like, and supplies the stream to the demultiplexer (DMUX) 1413 at apredetermined timing or based on a request from outside or the like.

In addition, the stream buffer 1414 buffers file data read from any ofvarious kinds of recording media via, for example, the connectivity 1321or the like, and supplies the data to the demultiplexer (DMUX) 1413 at apredetermined timing or based on a request from outside or the like.

Next, an example of an operation of the video processor 1332 having thisconfiguration will be described. For example, a video signal input tothe video processor 1332 from the connectivity 1321 or the like isconverted into digital image data in a predetermined format such as aYCbCr format of 4:2:2 of in the video input processing unit 1401, andsequentially written in the frame memory 1405. This digital image datais read by the first image enlarging and reducing unit 1402 or thesecond image enlarging and reducing unit 1403, undergoes formatconversion and an enlarging or reducing process in a predeterminedformat such as a YCbCr format of 4:2:0, and then is written in the framememory 1405 again. This image data is encoded by the encoding/decodingengine 1407, and written in the video ES buffer 1408A as a video stream.

In addition, an audio signal input to the video processor 1332 from theconnectivity 1321 is encoded by the audio encoder 1410, and then writtenin the audio ES buffer 1409A as an audio stream.

The video stream of the video ES buffer 1408A and the audio stream ofthe audio ES buffer 1409A are read and multiplexed by the multiplexer(MUX) 1412 to be converted into a transport stream, file data, or thelike. The transport stream generated by the multiplexer (MUX) 1412 isbuffered in the stream buffer 1414, and then output to an externalnetwork via, for example, the connectivity 1321, the broadband modem1333, or the like. In addition, the file data generated by themultiplexer (MUX) 1412 is buffered in the stream buffer 1414, and outputto, for example, the connectivity 1321 (of FIG. 29 ) to be recorded inany of various kinds of recording media.

In addition, a transport stream input to the video processor 1332 froman external network via, for example, the connectivity 1321, thebroadband modem 1333, or the like is buffered in the stream buffer 1414,and then demultiplexed by the demultiplexer (MUX) 1413. In addition, forexample, file data read from any of various kinds of recording media viathe connectivity 1321 and input to the video processor 1332 is bufferedin the stream buffer 1414, and then demultiplexed by the demultiplexer(DMUX) 1413. That is to say, the transport stream or the file data inputto the video processor 1332 is separated into a video stream and anaudio stream by the demultiplexer (DMUX) 1413.

The audio stream is supplied to the audio decoder 1411 via the audio ESbuffer 1409B to be decoded, and an audio signal is reproduced. Inaddition, the video stream is written in the video ES buffer 1408B, thensequentially read by the encoding/decoding engine 1407 to be decoded,and written in the frame memory 1405. The decoded image data undergoesan enlarging and reducing process by the second image enlarging andreducing unit 1403, and is written in the frame memory 1405. Then, thedecoded image data is read by the video output processing unit 1404,undergoes format conversion in a predetermined format such as the YCbCrformat of 4:2:2, and is further converted into an analog signal, and avideo signal is reproduced to be output.

When the present technology is applied to the video processor 1332configured in this manner, the present technology according to eachembodiment described above may be applied to the encoding/decodingengine 1407. That is, for example, the encoding/decoding engine 1407 mayhave functions of the image coding device 100 and the image decodingdevice 200 according to the above-described embodiment. In this manner,the video processor 1332 can obtain the same effects that were describedabove with reference to FIGS. 1 to 42 .

Also, in the encoding/decoding engine 1407, the present technology (thatis, functions of the image coding device and the image decoding deviceaccording to each embodiment described above) may be implemented byeither or both of hardware such as a logic circuit and software such asan embedded program.

<Other Example of a Configuration of a Video Processor>

FIG. 56 illustrates another example of a schematic configuration of thevideo processor 1332 to which the present technology is applied. In thecase of the example of FIG. 56 , the video processor 1332 has functionsof encoding and decoding video data in a predetermined scheme.

More specifically, as illustrated in FIG. 56 , the video processor 1332includes a control unit 1511, a display interface 1512, a display engine1513, an image processing engine 1514, and an internal memory 1515. Thevideo processor 1332 includes a codec engine 1516, a memory interface1517, a multiplexing and demultiplexing unit (MUX DMUX) 1518, a networkinterface 1519, and a video interface 1520.

The control unit 1511 controls an operation of each processing unit inthe video processor 1332, such as the display interface 1512, thedisplay engine 1513, the image processing engine 1514, and the codecengine 1516.

As illustrated in FIG. 56 , for example, the control unit 1511 includesa main CPU 1531, a sub-CPU 1532, and a system controller 1533. The mainCPU 1531 executes a program or the like to control an operation of eachprocessing unit in the video processor 1332. The main CPU 1531 generatesa control signal according to the program or the like and supplies thecontrol signal to each processing unit (that is, controls the operationof each processing unit). The sub-CPU 1532 serves as an auxiliary roleof the main CPU 1531. For example, the sub-CPU 1532 executes anoffspring process or a sub-routine of a program or the like executed bythe main CPU 1531. The system controller 1533 controls operations of themain CPU 1531 and the sub-CPU 1532, for example, designates programsexecuted by the main CPU 1531 and the sub-CPU 1532.

The display interface 1512 outputs the image data to, for example, theconnectivity 1321 under the control of the control unit 1511. Forexample, the display interface 1512 converts the image data of digitaldata into an analog signal and outputs the image data as the reproducedvideo signal or the image data of the digital data to a monitor deviceor the like of the connectivity 1321.

The display engine 1513 performs various conversion processes such asformat conversion, size conversion, and color gamut conversion on theimage data to match a hardware specification of the monitor device orthe like displaying the image under the control of the control unit1511.

The image processing engine 1514 performs predetermined image processingsuch as filter processing on the image data, for example, to improveimage quality under the control of the control unit 1511.

The internal memory 1515 is a memory shared by the display engine 1513,the image processing engine 1514, and the codec engine 1516 and providedinside the video processor 1332. For example, the internal memory 1515is used to transmit and receive data among the display engine 1513, theimage processing engine 1514, and the codec engine 1516. For example,the internal memory 1515 stores data supplied from the display engine1513, the image processing engine 1514, or the codec engine 1516 andsupplies the data to the display engine 1513, the image processingengine 1514, or the codec engine 1516, as necessary (for example,according to a request). The internal memory 1515 may be realized by anystorage device, but the internal memory 1515 is generally used to storedata with a small capacity such as parameters or image data in units ofblocks in many cases. Therefore, the internal memory 1515 is preferablyrealized by, for example, a semiconductor memory with a relatively smallcapacity (compared to, for example, the external memory 1312) and a fastresponse speed, such as a static random access memory (SRAM).

The codec engine 1516 performs a process related to encoding or decodingof the image data. Any encoding and decoding schemes to which the codecengine 1516 corresponds can be used, and the number of schemes may besingular or plural. For example, the codec engine 1516 may include codecfunctions of a plurality of encoding and decoding schemes, and mayencode the image data using the codec function selected therefrom anddecode the encoded data.

In the example illustrated in FIG. 56 , as functional blocks ofprocesses related to the codec, the codec engine 1516 includes, forexample, an MPEG-2 video 1541, an AVC/H.264 1542, an HEVC/H.265 1543, anHEVC/H.265 (scalable) 1544, and an HEVC/H.265 (multi-view) 1545 andincludes an MPEG-DASH 1551.

The MPEG-2 video 1541 is a functional block that encodes or decodes theimage data in an MPEG-2 scheme. The AVC/H.264 1542 is a functional blockthat encodes or decodes the image data in an AVC scheme. The HEVC/H.2651543 is a functional block that encodes or decodes the image data in anHEVC scheme. The HEVC/H.265 (scalable) 1544 is a functional block thatperforms scalable encoding or scalable decoding on the image data in anHEVC scheme. The HEVC/H.265 (multi-view) 1545 is a functional block thatperforms multi-view encoding or multi-view decoding on the image data inan HEVC scheme.

The MPEG-DASH 1551 is a function block that transmits and receives imagedata using an MPEG-Dynamic Adaptive Streaming over HTTP (MPEG-DASH)scheme. The MPEG-DASH is a technique, for streaming a video using HyperText Transfer Protocol (HTTP), and has a characteristic in whichappropriate data is selected from among a plurality of previouslyprepared pieces of coded data having different resolutions in units ofsegments, and is transmitted. The MPEG-DASH 1551 generates a streamcompliant with a standard, controls transmission of the stream or thelike, and uses the MPEG-2 Video 1541 to the HEVC/H.265 (Multi-view) 1545described above for coding and decoding the image data.

The memory interface 1517 is an interface for the external memory 1312.The data supplied from the image processing engine 1514 or the codecengine 1516 is supplied to the external memory 1312 via the memoryinterface 1517. The data read from the external memory 1312 is suppliedto the video processor 1332 (the image processing engine 1514 or thecodec engine 1516) via the memory interface 1517.

The multiplexing and demultiplexing unit (MUX DMUX) 1518 multiplexes ordemultiplexes various kinds of data related to images such as imagedata, video signals, and bit streams of encoded data. Any multiplexingand demultiplexing methods can be used. For example, at the time ofmultiplexing, the multiplexing and demultiplexing unit (MUX DMUX) 1518can collect a plurality of pieces of data into one piece of data and canalso add predetermined header information or the like to the data. Atthe time of demultiplexing, the multiplexing and demultiplexing unit(MUX DMUX) 1518 divides one piece of data into a plurality of pieces ofdata and can also add predetermined header information or the like toeach of the pieces of divided data. That is, the multiplexing anddemultiplexing unit (MUX DMUX) 1518 can convert the format of the datathrough the multiplexing and the demultiplexing. For example, themultiplexing and demultiplexing unit (MUX DMUX) 1518 can convert datainto a transport stream which is a bit stream with a transmission formator data (file data) with a file format for recording by multiplexing thebit stream. Of course, the reverse conversion can also be performedthrough the demultiplexing.

The network interface 1519 is, for example, an interface for thebroadband modem 1333, the connectivity 1321, or the like. The videointerface 1520 is, for example, an interface for the connectivity 1321,the camera 1322, or the like.

Next, an example of an operation of the video processor 1332 will bedescribed. For example, when the transport stream is received from anexternal network via the connectivity 1321, the broadband modem 1333, orthe like, the transport stream is supplied to the multiplexing anddemultiplexing unit (MUX DMUX) 1518 via the network interface 1519 to bedemultiplexed, and then is decoded by the codec engine 1516. Forexample, the image data obtained through the decoding of the codecengine 1516 is subjected to predetermined image processing by the imageprocessing engine 1514, is subjected to predetermined conversion by thedisplay engine 1513, and is supplied to, for example, the connectivity1321 via the display interface 1512, and then the image is displayed ona monitor. For example, the image data obtained through the decoding ofthe codec engine 1516 is re-encoded by the codec engine 1516, ismultiplexed by the multiplexing and demultiplexing unit (MUX DMUX) 1518to be converted into file data, is output to, for example, theconnectivity 1321 via the video interface 1520, and is recorded invarious recording media.

Further, for example, the file data of the encoded data read from arecording medium (not illustrated) by the connectivity 1321 or the likeand obtained by encoding the image data is supplied to the multiplexingand demultiplexing unit (MUX DMUX) 1518 via the video interface 1520 tobe demultiplexed, and then is decoded by the codec engine 1516. Theimage data obtained through the decoding of the codec engine 1516 issubjected to predetermined image processing by the image processingengine 1514, is subjected to predetermined conversion by the displayengine 1513, and is supplied to, for example, the connectivity 1321 viathe display interface 1512, and then the image is displayed on amonitor. For example, the image data obtained through the decoding ofthe codec engine 1516 is re-encoded by the codec engine 1516, ismultiplexed by the multiplexing and demultiplexing unit MUX DMUX) 1518to be converted into a transport stream, is supplied to, for example,the connectivity 1321 or the broadband modem 1333 via the networkinterface 1519, and is transmitted to another device (not illustrated).

Transmission and reception of the image data or other data between theprocessing units in the video processor 1332 are performed using, forexample, the internal memory 1515 or the external memory 1312. The powermanagement module 1313 controls power supply to, for example, thecontrol unit 1511.

When the present technology is applied to the video processor 1332configured in this manner, the present technology according to eachembodiment described above may be applied to the codec engine 1516. Thatis, for example, the codec engine 1516 may have function blocks thatrealize the image coding device 100 and the image decoding device 200according to the above-described embodiment. In this manner, the videoprocessor 1332 can obtain the same effects that were described abovewith reference to FIGS. 1 to 42 .

Also, in the codec engine 1516, the present technology (that is,functions of the image coding device and the image decoding deviceaccording to each embodiment described above) may be implemented byeither or both of hardware such as a logic circuit and software such asan embedded program.

The two configurations of the video processor 1332 have beenexemplified, but the configuration of the video processor 1332 isarbitrary and may be a configuration other than the two configurationsdescribed above. The video processor 1332 may be configured as a singlesemiconductor chip or may be configured as a plurality of semiconductorchips. For example, a 3-dimensional laminated LSI in which a pluralityof semiconductors are laminated may be used. The video processor 1332may be realized by a plurality of LSIs.

<Application Examples to Devices>

The video set 1300 can be embedded in various devices that process imagedata. For example, the video set 1300 can be embedded in the televisiondevice 900 (FIG. 47 ), the mobile telephone 920 (FIG. 48 ), therecording and reproduction device 940 (FIG. 49 ), the imaging device 960(FIG. 50 ), or the like. By embedding the video set 1300, the device canobtain the same advantages as the advantages described with reference toFIGS. 1 to 42 .

In addition, the video set 1300 may be embedded in, for example, aterminal device such as the personal computer 1004, the AV instrument1005, the tablet device 1006, and the mobile telephone 1007 in the datatransmission system 1000 of FIG. 51 , the broadcast station 1101 and theterminal device 1102 in the data transmission system 1100 of FIG. 52 andthe imaging device 1201 and the scalably coded data storage device 1202in the imaging system 1200 of FIG. 53 . When the video set 1300 isembedded, the device can obtain the same effects that were describedabove with reference to FIGS. 1 to 42 .

A part of each configuration of the above-described video set 1300 canalso be implemented as a configuration to which the present technologyis applied, as long as the part of the configuration includes the videoprocessor 1332. For example, only the video processor 1332 can beimplemented as a video processor to which the present technology isapplied. For example, the video module 1331 or the processor indicatedby the dashed line 1341, as described above, can be implemented as aprocessor, a module, or the like to which the present technology isapplied. Further, for example, the video module 1311, the external 1312,the power management module 1313, and the frontend module 1314 can becombined to be implemented as a video unit 1361 to which the presenttechnology is applied. It is possible to obtain the same advantages asthe advantages described with reference to FIGS. 1 to 42 regardless ofthe configuration.

That is, any configuration can be embedded in various devices processingimage data, as in the case of the video set 1300, as long as theconfiguration includes the video processor 1332. For example, the videoprocessor 1332, the processor indicated by the dashed line 1341, thevideo module 1311, or the video unit 1361 may be embedded in thetelevision device 900 (FIG. 47 ), the mobile telephone 920 (FIG. 48 ),the recording and reproduction device 940 (FIG. 49 ), the imaging device960 (FIG. 50 ), terminal devices such as the personal computer 1004, theAV instrument 1005, the tablet device 1006, and the mobile telephone1007 in the data transmission system 1000 of FIG. 51 , the broadcaststation 1101 and the terminal device 1102 in the data transmissionsystem 1100 of FIG. 52 , and the imaging device 1201 and the scalablycoded data storage device 1202 in the imaging system 1200 of FIG. 53 .By embedding any configuration to which the present technology isapplied, the device can obtain the same advantages as the advantagesdescribed with reference to FIGS. 1 to 42 , as in the video set 1300.

In the present specification, the examples in which the various piecesof information are multiplexed in the coding stream and are transmittedfrom the encoding side to the decoding side have been described.However, the methods of transmitting the information are not limited tothe examples. For example, the information may be transmitted orrecorded as separate pieces of data associated with the coding bitstream without being multiplexed in the coding bit stream. Here, theterm “associated” means that an image (which may be a part of an image,such as a slice or a block) included in a bit stream and informationcorresponding to the image can be linked at the time of decoding. Thatis, the information may be transmitted along a different transmissionpath from the image (or the bit stream). The information may be recordedin a different recording medium (or a different recording area of thesame recording medium) from the image (or the bit stream). Further, theinformation and the image (or the bit stream) may be mutuallyassociated, for example, in any unit such as a plurality of frames, asingle frame, or a part of a frame.

Additionally, the present technology may also be configured as below.

(1)

An image coding device including:

-   -   a coding unit configured to code image data;    -   a decoding load definition information setting unit configured        to set decoding load definition information for defining a        magnitude of a load of a decoding process of an independently        decodable partial region of an image of the image data; and    -   a transmission unit configured to transmit coded data of the        image data generated by the coding unit and the decoding load        definition information set by the decoding load definition        information setting unit.

(2)

The image coding device according to any of (1), and (3) to (13),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process.

(3)

The image coding device according to any of (1), (2), and (4) to (13),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a size of the partial region.

(4)

The image coding device according to any of to (3), and (5) to (13),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a length in a vertical direction and information        indicating a length in a horizontal direction of the partial        region.

(5)

The image coding device according to any of (1) to (4), and (6) to (13),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a maximum input bit rate and a buffer capacity of a        virtual reference decoder configured to decode the partial        region.

(6)

The image coding device according to any of (1) to (5), and (7) to (13),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process common in a        plurality of the partial regions.

(7)

The image coding device according to any of (1) to (6), and (8) to (13),

-   -   wherein the decoding load definition information includes        information for defining a size in a vertical direction and a        size in a horizontal direction of the partial region        corresponding to each level indicating a magnitude of a load of        the decoding process.

(8)

The image coding device according to any of (1) to (7), and (9) to (13),

-   -   wherein the decoding load definition information includes        information for defining a maximum value of a level indicating a        magnitude of a load of the decoding process in the image.

(9)

The image coding device according to any of (1) to (8), and (10) to(13),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process, and    -   wherein the decoding load definition information setting unit        defines a parameter of the level for a predetermined region        including the partial region and assigns it to definition of the        image unit.

(10)

The image coding device according to any of (1) to (9), and (11) to(13),

-   -   wherein the decoding load definition information setting unit        sets the decoding load definition information for each of the        partial regions in supplemental enhancement information (SEI) of        the independently decodable partial region.

(11)

The image coding device according to any of (1) to (10), (12), and (13),

-   -   wherein the image data includes a plurality of layers, and    -   wherein the decoding load definition information setting unit        sets the decoding load definition information of the plurality        of layers in the SEI.

(12)

The image coding device according to any of (1) to (11), and (13),

-   -   wherein the decoding load definition information setting unit        further sets information indicating whether the decoding load        definition information is set in the SEI or the same decoding        load definition information as the decoding load definition        information set in the SEI in a sequence parameter set (SPS).

(13)

The image coding device according to any of (1) to (12),

-   -   wherein the decoding load definition information includes        information indicating a size of the partial region serving as a        reference, and a level indicating a magnitude of a load of a        decoding process of the partial region.

(14)

An image coding method including:

-   -   coding image data;    -   setting decoding load definition information for defining a        magnitude of a load of a decoding process of an independently        decodable partial region of an image of the image data; and    -   transmitting generated coded data of the image data and the set        decoding load definition information.

(15)

An image decoding device including:

-   -   an acquisition unit configured to acquire coded data of image        data and decoding load definition information for defining a        magnitude of a load of a decoding process of a partial region of        an image of the image data, the partial region being        independently decodable;    -   an analysis unit configured to analyze the decoding load        definition information acquired by the acquisition unit;    -   a control unit configured to control decoding of the coded data        acquired by the acquisition unit based on an analysis result of        the decoding load definition information by the acquisition        unit; and    -   a decoding unit configured to decode the coded data acquired by        the acquisition unit under control of the control unit.

(16)

The image decoding device according to any of (15), and (17) to (27),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process.

(17)

The image decoding device according to any of (15), (16), and (18) to(27),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a size of the partial region.

(18)

The image decoding device according to any of (15) to (17), and (19) to(27),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a length in a vertical direction and information        indicating a length in a horizontal direction of the partial        region.

(19)

The image decoding device according to any of (15) to (18), and (20) to(27),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a maximum input bit rate and a buffer capacity of a        virtual reference decoder configured to decode the partial        region.

(20)

The image decoding device according to any of (15) to (19), and (21) to(27),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process common in a        plurality of the partial regions.

(21)

The image decoding device according to any of (15) to (20), and (27),

-   -   wherein the decoding load definition information includes        information for defining a size in a vertical direction and a        size in a horizontal direction of the partial region        corresponding to each level indicating a magnitude of a load of        the decoding process.

(22)

The image decoding device according to any of (15) to (21) and (23) to(27),

-   -   wherein the decoding load definition information includes        information for defining a maximum value of a level indicating a        magnitude of a load of the decoding process in the image.

(23)

The image decoding device according to any of (15) to (22), and (24) to(27),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process, and    -   wherein the control unit controls decoding the coding data using        a parameter of the level defined for a predetermined region        including the partial region and assigned to definition of the        image unit

(24)

The image decoding device according to any of (15) to (23), and (25) to(27),

-   -   wherein the analysis unit analyzes the decoding load definition        information set for each of the partial regions in supplemental        enhancement information (SEI) of the independently decodable        partial region.

(25)

The image decoding device according to any of (15) to (24), (26), and(27),

-   -   wherein the image data includes a plurality of layers, and    -   wherein the analysis unit analyzes the decoding load definition        information of the plurality of layers set in the SEI.

(26)

The image decoding device according to any of (15) to (25), and (27),

-   -   wherein the analysis unit further analyzes information        indicating whether the decoding load definition information is        set in the SEI or the same decoding load definition information        as the decoding load definition information set in the SEI set        in a sequence parameter set (SPS).

(27)

The image decoding device according to any of (15) to (26),

-   -   wherein the decoding load definition information includes        information indicating a size of the partial region serving as a        reference, and a level indicating a magnitude of a load of a        decoding process of the partial region.

(28)

An image decoding method including:

-   -   acquiring coded data of image data and decoding load definition        information for defining a magnitude of a load of a decoding        process of a partial region of an image of the image data;    -   controlling decoding of the acquired coded data based on the        acquired decoding load definition information; and    -   decoding the acquired coded data according to the controlling.

(31)

An image decoding device including:

-   -   an acquisition unit configured to acquire coded data of image        data and decoding load definition information for defining a        magnitude of a load of a decoding process of a partial region of        an image of the image data;    -   a control unit configured to control decoding of the coded data        acquired by the acquisition unit based on the decoding load        definition information acquired by the acquisition unit; and    -   a decoding unit configured to decode the coded data acquired by        the acquisition unit under control of the control unit.

(32)

The image decoding device according to (31),

-   -   wherein the partial region is independently decodable.

(33)

The image decoding device according to (31) or (32),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to a level indicating a        magnitude of a load of the decoding process.

(34)

The image decoding device according to any of (31) to (33),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a size of the partial region.

(35)

The image decoding device according to any of (31) to (34),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load of a decoding        process of the partial region according to information        indicating a length in a vertical direction and information        indicating a length in a horizontal direction of the partial        region.

(36)

The image decoding device according to any of (31) to (35),

-   -   wherein the decoding load definition information is included in        supplemental enhancement information (SEI) of an independently        decodable partial region.

(37)

The image decoding device according to any of (31) to (36),

-   -   wherein the image data includes a plurality of layers, and    -   wherein the decoding load definition information of the        plurality of layers is included in the SEI.

(38)

The image decoding device according to any of (31) to (37),

-   -   wherein the decoding load definition information includes        information indicating a size of the partial region serving as a        reference, and a level indicating a magnitude of a load of a        decoding process of the partial region.

(39)

The image decoding device according to any of (31) to (38),

-   -   wherein the partial region is a tile.

(40)

The image decoding device according to any of (3!) to (39),

-   -   wherein the partial region is a set of a plurality of tiles.

(41)

The image decoding device according to any of (31) to (40),

-   -   wherein the decoding load definition information includes        information for defining a maximum magnitude of a load of a        decoding process among a plurality of partial regions included        in a picture of the image data according to a level indicating a        magnitude of a load of the decoding process.

(42)

The image decoding device according to any of (31) to (41),

-   -   wherein the decoding load definition information includes        information for defining a magnitude of a load common in a        plurality of partial regions included in a picture of the image        data according to a level indicating a magnitude of a load of        the decoding process.

(43)

The image decoding device according to any of (31) to (42),

-   -   wherein, when the plurality of partial regions included in the        picture have an L shape, a magnitude of the load is defined for        a rectangular region including the L shape.

(44)

The image decoding device according to any of (31) to (43),

-   -   wherein the acquisition unit further acquires information        indicating whether the decoding load definition information is        set, and when the acquired information indicates that the        decoding load definition information is set, acquires the the        decoding load definition information.

(45)

An image decoding method including:

-   -   acquiring coded data of image data and decoding load definition        information for defining a magnitude of a load of a decoding        process of a partial region of an image of the image data;    -   controlling decoding of the acquired coded data based on the        acquired decoding load definition information; and    -   decoding the acquired coded data according to the controlling.

REFERENCE SIGNS LIST

-   -   100 image coding device    -   101 base layer image coding unit    -   102 enhancement layer image coding unit    -   103 multiplexing unit    -   104 control unit    -   128 header information generating unit    -   148 header information generating unit    -   151 decoding load related information acquisition unit    -   152 MCTS SEI generating unit    -   153 SPS generating unit    -   200 image decoding device    -   201 demultiplexing unit    -   202 base layer image decoding unit    -   203 enhancement layer image decoding unit    -   204 control unit    -   224 header information analyzing unit    -   244 header information analyzing unit    -   251 header information acquisition unit    -   252 SPS analyzing unit    -   253 MCTS SEI analyzing unit    -   254 level specifying unit    -   255 providing unit

The invention claimed is:
 1. An image coding device comprising: a codingunit configured to code an image; and a setting unit configured to setinformation indicating a level indicating a load of a decoding processof an entirety of the image, a size of a partial region of the image,and a level indicating a load of a decoding process of the partialregion, for controlling decoding of coded data, which is transmittedfrom the image coding device to an image decoding device and in whichthe image is coded, the information indicating the size of the partialregion of the image and the level indicating the load of the decodingprocess of the partial region both also being transmitted from the imagecoding device together with the coded data to the image decoding device,wherein the partial region is an independently decodable region of theimage that is less than the entirety of the image, and the setting unitis further configured to set the information indicating the levelindicating the load of the decoding process of the entirety of the imageand the level indicating the load of the decoding process of the partialregion by setting a first level indicating the load of the decodingprocess of the entirety of the image and setting a second levelindicating the load of the decoding process of the partial region, thefirst level being different than the second level, and wherein thecoding unit and the setting unit are each implemented via at least oneprocessor.
 2. The image coding device according to claim 1, wherein thesetting unit generates information for defining the load of the decodingprocess of the partial region according to the level indicating the loadof the decoding process of the partial region.
 3. The image codingdevice according to claim 1, wherein the setting unit generatesinformation for defining the load of the decoding process of the partialregion according to information indicating the size of the partialregion.
 4. The image coding device according to claim 1, wherein thesetting unit generates information for defining the load of the decodingprocess of the partial region according to information indicating alength in a vertical direction and information indicating a length in ahorizontal direction of the partial region.
 5. The image coding deviceaccording to claim 1, further comprising a transmission unit configuredto transmit information indicating the size of the partial region andthe level indicating the load of the decoding process of the partialregion as decoding load definition information indicating the load ofthe decoding process of the partial region, wherein the transmissionunit is implemented via at least one processor.
 6. The image codingdevice according to claim 5, wherein the transmission unit transmits thedecoding load definition information as auxiliary information of codeddata in which the image obtained by the coding unit is coded.
 7. Theimage coding device according to claim 6, wherein the transmission unittransmits the decoding load definition information as supplementalenhancement information (SEI) of an independently decodable partialregion.
 8. The image coding device according to claim 7, wherein thecoded data includes image data of a plurality of layers, and wherein thetransmission unit transmits the decoding load definition information ofthe plurality of layers as the SEI.
 9. The image coding device accordingto claim 1, wherein the partial region is a tile.
 10. The image codingdevice according to claim 1, wherein the partial region is a set of aplurality of tiles.
 11. The image coding device according to claim 1,wherein the setting unit generates information for defining a maximummagnitude of a load of a decoding process among a plurality of partialregions included in a picture of the image data included in the codeddata according to a level indicating a load of the decoding process. 12.The image coding device according to claim 1, wherein the setting unitgenerates information for defining a load common in a plurality ofpartial regions included in a picture of the image data included in thecoded data according to a level indicating a load of the decodingprocess.
 13. The image coding device according to claim 12, wherein,when a plurality of the partial regions included in the picture have anL shape, the setting unit generates information for defining the loadfor a rectangular region including the L shape.
 14. The image codingdevice according to claim 1, wherein the setting unit generatesinformation indicating whether the decoding load definition informationis set.
 15. An image coding method comprising: coding an image; andsetting information indicating a level indicating a load of a decodingprocess of an entirety of the image, a size of a partial region of theimage, and a level indicating a load of a decoding process of thepartial region, for controlling decoding of coded data, which istransmitted from the image coding device to an image decoding device andin which the image is coded, the information indicating the size of thepartial region of the image and the level indicating the load of thedecoding process of the partial region both also being transmitted fromthe image coding device together with the coded data to the imagedecoding device, wherein the partial region is an independentlydecodable region of the image that is less than the entirety of theimage, and the setting of information indicating the level indicatingthe load of the decoding process of the entirety of the image and thelevel indicating the load of the decoding process of the partial regionincludes setting a first level indicating the load of the decodingprocess of the entirety of the image and setting a second levelindicating the load of the decoding process of the partial region, thefirst level being different than the second level.
 16. The image codingdevice according to claim 1, wherein the information indicating the sizeof the partial region of the image and the level indicating the load ofthe decoding process of the partial region are both received togetherwith the coded data by the image decoding device.
 17. The image codingdevice according to claim 1, wherein the information indicating the sizeof the partial region of the image includes a number of tiles of aplurality of tiles obtained by uniformly dividing the entire image. 18.A non-transitory computer-readable medium having embodied thereon aprogram, which when executed by a computer causes the computer toexecute an image coding method, the method comprising: coding an image;and setting information indicating a level indicating a load of adecoding process of an entirety of the image, a size of a partial regionof the image, and a level indicating a load of a decoding process of thepartial region, for controlling decoding of coded data, which istransmitted from the image coding device to an image decoding device andin which the image is coded, the information indicating the size of thepartial region of the image and the level indicating the load of thedecoding process of the partial region both also being transmitted fromthe image coding device together with the coded data to the imagedecoding device, wherein the partial region is an independentlydecodable region of the image that is less than the entirety of theimage, and the setting of information indicating the level indicatingthe load of the decoding process of the entirety of the image and thelevel indicating the load of the decoding process of the partial regionincludes setting a first level indicating the load of the decodingprocess of the entirety of the image and setting a second levelindicating the load of the decoding process of the partial region, thefirst level being different than the second level.